INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) INFO: 172.19.0.1:48090 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:37602 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:42136 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:48514 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:39136 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:50130 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:50144 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:50160 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:57732 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:39776 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:37920 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:33894 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:56688 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:33718 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:33720 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:33730 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:35134 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:18:30,921 - lava_callback - INFO - Received callback for node 692ab261d481a0c6f8cb4d84 2025-11-30 15:18:30,951 - lava_callback - INFO - Starting async job submit for node 692ab261d481a0c6f8cb4d84 INFO: 172.19.0.1:45718 - "POST /node/692ab261d481a0c6f8cb4d84 HTTP/1.0" 202 Accepted 2025-11-30 15:18:31,277 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:18:31,291 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab261d481a0c6f8cb4d84 HTTP/1.1" 502 150 2025-11-30 15:18:31,291 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ab261d481a0c6f8cb4d84'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:18:31,291 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ab261d481a0c6f8cb4d84 2025-11-30 15:18:31,292 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab261d481a0c6f8cb4d84 HTTP/1.1" 502 150 2025-11-30 15:18:31,292 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ab261d481a0c6f8cb4d84'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:18:33,292 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ab261d481a0c6f8cb4d84 2025-11-30 15:18:33,294 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab261d481a0c6f8cb4d84 HTTP/1.1" 502 150 2025-11-30 15:18:33,294 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ab261d481a0c6f8cb4d84'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:45720 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:18:37,294 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ab261d481a0c6f8cb4d84 2025-11-30 15:18:37,295 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab261d481a0c6f8cb4d84 HTTP/1.1" 502 150 2025-11-30 15:18:37,295 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ab261d481a0c6f8cb4d84'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:18:45,295 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ab261d481a0c6f8cb4d84 2025-11-30 15:18:45,297 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab261d481a0c6f8cb4d84 HTTP/1.1" 502 150 2025-11-30 15:18:45,297 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ab261d481a0c6f8cb4d84'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:56126 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:18:50,892 - lava_callback - INFO - Received callback for node 692ab276d481a0c6f8cb4da6 2025-11-30 15:18:50,898 - lava_callback - INFO - Starting async job submit for node 692ab276d481a0c6f8cb4da6 INFO: 172.19.0.1:35698 - "POST /node/692ab276d481a0c6f8cb4da6 HTTP/1.0" 202 Accepted 2025-11-30 15:18:50,924 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:18:50,964 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab276d481a0c6f8cb4da6 HTTP/1.1" 502 150 2025-11-30 15:18:50,964 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ab276d481a0c6f8cb4da6'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:18:50,964 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ab276d481a0c6f8cb4da6 2025-11-30 15:18:50,964 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab276d481a0c6f8cb4da6 HTTP/1.1" 502 150 2025-11-30 15:18:50,964 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ab276d481a0c6f8cb4da6'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:18:52,965 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ab276d481a0c6f8cb4da6 2025-11-30 15:18:52,969 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab276d481a0c6f8cb4da6 HTTP/1.1" 502 150 2025-11-30 15:18:52,969 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ab276d481a0c6f8cb4da6'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:35712 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:18:56,969 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ab276d481a0c6f8cb4da6 2025-11-30 15:18:56,973 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab276d481a0c6f8cb4da6 HTTP/1.1" 502 150 2025-11-30 15:18:56,973 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ab276d481a0c6f8cb4da6'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:19:01,297 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ab261d481a0c6f8cb4d84 2025-11-30 15:19:01,298 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab261d481a0c6f8cb4d84 HTTP/1.1" 502 150 2025-11-30 15:19:01,298 - lava_callback - ERROR - Error processing callback for node 692ab261d481a0c6f8cb4d84: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ab261d481a0c6f8cb4d84 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ab261d481a0c6f8cb4d84 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ab261d481a0c6f8cb4d84 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 15:19:04,974 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ab276d481a0c6f8cb4da6 2025-11-30 15:19:04,975 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab276d481a0c6f8cb4da6 HTTP/1.1" 502 150 2025-11-30 15:19:04,975 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ab276d481a0c6f8cb4da6'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:51052 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:53246 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:53258 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:53260 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:19:20,976 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ab276d481a0c6f8cb4da6 2025-11-30 15:19:20,977 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab276d481a0c6f8cb4da6 HTTP/1.1" 502 150 2025-11-30 15:19:20,977 - lava_callback - ERROR - Error processing callback for node 692ab276d481a0c6f8cb4da6: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ab276d481a0c6f8cb4da6 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ab276d481a0c6f8cb4da6 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ab276d481a0c6f8cb4da6 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:50146 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:45610 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:42436 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:19:47,252 - lava_callback - INFO - Received callback for node 692ab278d481a0c6f8cb4da9 INFO: 172.19.0.1:42442 - "POST /node/692ab278d481a0c6f8cb4da9 HTTP/1.0" 202 Accepted 2025-11-30 15:19:47,259 - lava_callback - INFO - Starting async job submit for node 692ab278d481a0c6f8cb4da9 2025-11-30 15:19:47,292 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:19:47,324 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab278d481a0c6f8cb4da9 HTTP/1.1" 502 150 2025-11-30 15:19:47,324 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ab278d481a0c6f8cb4da9'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:19:47,324 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ab278d481a0c6f8cb4da9 2025-11-30 15:19:47,325 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab278d481a0c6f8cb4da9 HTTP/1.1" 502 150 2025-11-30 15:19:47,325 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ab278d481a0c6f8cb4da9'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:19:49,325 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ab278d481a0c6f8cb4da9 2025-11-30 15:19:49,326 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab278d481a0c6f8cb4da9 HTTP/1.1" 502 150 2025-11-30 15:19:49,326 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ab278d481a0c6f8cb4da9'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:19:53,326 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ab278d481a0c6f8cb4da9 2025-11-30 15:19:53,328 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab278d481a0c6f8cb4da9 HTTP/1.1" 502 150 2025-11-30 15:19:53,328 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ab278d481a0c6f8cb4da9'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:39164 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:20:01,328 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ab278d481a0c6f8cb4da9 2025-11-30 15:20:01,330 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab278d481a0c6f8cb4da9 HTTP/1.1" 502 150 2025-11-30 15:20:01,330 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ab278d481a0c6f8cb4da9'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:35120 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:48962 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:48974 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:48980 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:20:17,330 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ab278d481a0c6f8cb4da9 2025-11-30 15:20:17,331 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab278d481a0c6f8cb4da9 HTTP/1.1" 502 150 2025-11-30 15:20:17,331 - lava_callback - ERROR - Error processing callback for node 692ab278d481a0c6f8cb4da9: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ab278d481a0c6f8cb4da9 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ab278d481a0c6f8cb4da9 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ab278d481a0c6f8cb4da9 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:47172 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:40760 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:33082 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:45448 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:50058 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:21:10,847 - lava_callback - INFO - Received callback for node 692ab274d481a0c6f8cb4da3 INFO: 172.19.0.1:49304 - "POST /node/692ab274d481a0c6f8cb4da3 HTTP/1.0" 202 Accepted 2025-11-30 15:21:10,853 - lava_callback - INFO - Starting async job submit for node 692ab274d481a0c6f8cb4da3 2025-11-30 15:21:10,930 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:21:10,940 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab274d481a0c6f8cb4da3 HTTP/1.1" 502 150 2025-11-30 15:21:10,940 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ab274d481a0c6f8cb4da3'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:21:10,941 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ab274d481a0c6f8cb4da3 2025-11-30 15:21:10,941 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab274d481a0c6f8cb4da3 HTTP/1.1" 502 150 2025-11-30 15:21:10,941 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ab274d481a0c6f8cb4da3'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:49312 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:21:12,941 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ab274d481a0c6f8cb4da3 2025-11-30 15:21:12,943 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab274d481a0c6f8cb4da3 HTTP/1.1" 502 150 2025-11-30 15:21:12,943 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ab274d481a0c6f8cb4da3'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:49328 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:49342 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:21:16,943 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ab274d481a0c6f8cb4da3 2025-11-30 15:21:16,944 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab274d481a0c6f8cb4da3 HTTP/1.1" 502 150 2025-11-30 15:21:16,944 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ab274d481a0c6f8cb4da3'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:21:24,945 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ab274d481a0c6f8cb4da3 2025-11-30 15:21:24,946 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab274d481a0c6f8cb4da3 HTTP/1.1" 502 150 2025-11-30 15:21:24,946 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ab274d481a0c6f8cb4da3'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:52454 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:36700 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:21:40,946 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ab274d481a0c6f8cb4da3 2025-11-30 15:21:40,947 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab274d481a0c6f8cb4da3 HTTP/1.1" 502 150 2025-11-30 15:21:40,947 - lava_callback - ERROR - Error processing callback for node 692ab274d481a0c6f8cb4da3: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ab274d481a0c6f8cb4da3 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ab274d481a0c6f8cb4da3 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ab274d481a0c6f8cb4da3 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:34184 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:51084 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:55170 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:38186 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:38200 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:38202 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:36218 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:22:31,273 - lava_callback - INFO - Received callback for node 692ab263d481a0c6f8cb4d87 INFO: 172.19.0.1:50158 - "POST /node/692ab263d481a0c6f8cb4d87 HTTP/1.0" 202 Accepted 2025-11-30 15:22:31,282 - lava_callback - INFO - Starting async job submit for node 692ab263d481a0c6f8cb4d87 2025-11-30 15:22:31,421 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:22:31,431 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab263d481a0c6f8cb4d87 HTTP/1.1" 502 150 2025-11-30 15:22:31,431 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ab263d481a0c6f8cb4d87'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:22:31,431 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ab263d481a0c6f8cb4d87 2025-11-30 15:22:31,432 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab263d481a0c6f8cb4d87 HTTP/1.1" 502 150 2025-11-30 15:22:31,432 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ab263d481a0c6f8cb4d87'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:22:33,432 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ab263d481a0c6f8cb4d87 2025-11-30 15:22:33,433 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab263d481a0c6f8cb4d87 HTTP/1.1" 502 150 2025-11-30 15:22:33,433 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ab263d481a0c6f8cb4d87'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:50174 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:22:37,434 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ab263d481a0c6f8cb4d87 2025-11-30 15:22:37,442 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab263d481a0c6f8cb4d87 HTTP/1.1" 502 150 2025-11-30 15:22:37,442 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ab263d481a0c6f8cb4d87'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:22:45,442 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ab263d481a0c6f8cb4d87 2025-11-30 15:22:45,443 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab263d481a0c6f8cb4d87 HTTP/1.1" 502 150 2025-11-30 15:22:45,443 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ab263d481a0c6f8cb4d87'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:52178 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:33392 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:23:01,444 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ab263d481a0c6f8cb4d87 2025-11-30 15:23:01,445 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ab263d481a0c6f8cb4d87 HTTP/1.1" 502 150 2025-11-30 15:23:01,445 - lava_callback - ERROR - Error processing callback for node 692ab263d481a0c6f8cb4d87: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ab263d481a0c6f8cb4d87 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ab263d481a0c6f8cb4d87 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ab263d481a0c6f8cb4d87 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:48762 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:34452 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:34458 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:34470 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:35876 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:46040 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:48498 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:57180 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:60228 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:52166 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:52172 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:52174 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:60358 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:40114 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:46884 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:41270 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:35692 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:37054 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:37058 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:37062 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:35150 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:42986 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:42276 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:50294 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:38668 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:36936 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:36948 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:36952 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:46016 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:54966 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:48424 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:26:52,063 - lava_callback - INFO - Received callback for node 6929c552498c72996de1fc3d INFO: 172.19.0.1:41134 - "POST /node/6929c552498c72996de1fc3d HTTP/1.0" 202 Accepted 2025-11-30 15:26:52,085 - lava_callback - INFO - Starting async job submit for node 6929c552498c72996de1fc3d 2025-11-30 15:26:53,541 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:26:53,552 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c552498c72996de1fc3d HTTP/1.1" 502 150 2025-11-30 15:26:53,552 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c552498c72996de1fc3d'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:26:53,552 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c552498c72996de1fc3d 2025-11-30 15:26:53,552 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c552498c72996de1fc3d HTTP/1.1" 502 150 2025-11-30 15:26:53,553 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c552498c72996de1fc3d'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:26:55,553 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c552498c72996de1fc3d 2025-11-30 15:26:55,554 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c552498c72996de1fc3d HTTP/1.1" 502 150 2025-11-30 15:26:55,554 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c552498c72996de1fc3d'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:41142 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:26:59,554 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c552498c72996de1fc3d 2025-11-30 15:26:59,555 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c552498c72996de1fc3d HTTP/1.1" 502 150 2025-11-30 15:26:59,555 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c552498c72996de1fc3d'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:51270 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:27:07,556 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c552498c72996de1fc3d 2025-11-30 15:27:07,557 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c552498c72996de1fc3d HTTP/1.1" 502 150 2025-11-30 15:27:07,557 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c552498c72996de1fc3d'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:51064 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:51070 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:51080 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:27:23,557 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c552498c72996de1fc3d 2025-11-30 15:27:23,558 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c552498c72996de1fc3d HTTP/1.1" 502 150 2025-11-30 15:27:23,558 - lava_callback - ERROR - Error processing callback for node 6929c552498c72996de1fc3d: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c552498c72996de1fc3d (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c552498c72996de1fc3d (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c552498c72996de1fc3d (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:45268 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:42030 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:47358 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:51524 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:45678 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:28:10,918 - lava_callback - INFO - Received callback for node 6929c58d498c72996de1fc7a INFO: 172.19.0.1:52876 - "POST /node/6929c58d498c72996de1fc7a HTTP/1.0" 202 Accepted 2025-11-30 15:28:10,924 - lava_callback - INFO - Starting async job submit for node 6929c58d498c72996de1fc7a 2025-11-30 15:28:10,946 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:28:10,965 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c58d498c72996de1fc7a HTTP/1.1" 502 150 2025-11-30 15:28:10,965 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c58d498c72996de1fc7a'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:28:10,965 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c58d498c72996de1fc7a 2025-11-30 15:28:10,965 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c58d498c72996de1fc7a HTTP/1.1" 502 150 2025-11-30 15:28:10,966 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c58d498c72996de1fc7a'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:52890 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:28:12,966 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c58d498c72996de1fc7a 2025-11-30 15:28:12,967 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c58d498c72996de1fc7a HTTP/1.1" 502 150 2025-11-30 15:28:12,967 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c58d498c72996de1fc7a'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:52904 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:52914 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:28:16,967 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c58d498c72996de1fc7a 2025-11-30 15:28:16,968 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c58d498c72996de1fc7a HTTP/1.1" 502 150 2025-11-30 15:28:16,968 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c58d498c72996de1fc7a'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:28:24,969 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c58d498c72996de1fc7a 2025-11-30 15:28:24,970 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c58d498c72996de1fc7a HTTP/1.1" 502 150 2025-11-30 15:28:24,970 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c58d498c72996de1fc7a'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:43614 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:51074 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:28:40,970 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c58d498c72996de1fc7a 2025-11-30 15:28:40,971 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c58d498c72996de1fc7a HTTP/1.1" 502 150 2025-11-30 15:28:40,972 - lava_callback - ERROR - Error processing callback for node 6929c58d498c72996de1fc7a: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c58d498c72996de1fc7a (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c58d498c72996de1fc7a (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c58d498c72996de1fc7a (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:47046 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:54088 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:41188 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:55210 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:55222 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:55230 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:45832 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:34574 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:55894 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:41266 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:38424 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:54396 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:54402 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:54410 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:49938 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:50072 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:42090 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:37898 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:52140 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:56038 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:56046 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:56054 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:53206 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:49314 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:59722 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:47500 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:41216 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:52912 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:52924 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:52934 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:37686 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:53318 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:47118 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:33776 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:59148 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:55252 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:55268 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:55274 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:56970 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:35418 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:53402 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:40228 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:57264 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:38236 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:38244 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:38256 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:59170 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:60550 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:33440 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:60078 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:36268 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:41556 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:41570 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:41578 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:52040 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:50866 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:46496 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:39034 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:37532 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:55798 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:55806 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:55816 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:56736 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:53234 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:43518 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:37542 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:48416 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:35452 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:35468 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:35478 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:35444 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:51562 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:40874 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:36466 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:57570 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:50768 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:50782 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:50784 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:50934 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:41198 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:50520 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:54630 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:57916 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:38146 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:38150 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:38160 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:43320 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:35158 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:59918 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:39980 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:34414 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:40:07,749 - lava_callback - INFO - Received callback for node 692aca88d481a0c6f8cb588a INFO: 172.19.0.1:34422 - "POST /node/692aca88d481a0c6f8cb588a HTTP/1.0" 202 Accepted 2025-11-30 15:40:07,759 - lava_callback - INFO - Starting async job submit for node 692aca88d481a0c6f8cb588a 2025-11-30 15:40:07,781 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:40:07,797 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca88d481a0c6f8cb588a HTTP/1.1" 502 150 2025-11-30 15:40:07,797 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca88d481a0c6f8cb588a'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:40:07,797 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca88d481a0c6f8cb588a 2025-11-30 15:40:07,798 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca88d481a0c6f8cb588a HTTP/1.1" 502 150 2025-11-30 15:40:07,798 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca88d481a0c6f8cb588a'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:40:09,799 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca88d481a0c6f8cb588a 2025-11-30 15:40:09,800 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca88d481a0c6f8cb588a HTTP/1.1" 502 150 2025-11-30 15:40:09,800 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca88d481a0c6f8cb588a'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:58958 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:40:13,800 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca88d481a0c6f8cb588a 2025-11-30 15:40:13,801 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca88d481a0c6f8cb588a HTTP/1.1" 502 150 2025-11-30 15:40:13,801 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca88d481a0c6f8cb588a'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:58962 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:58968 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:40:21,802 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca88d481a0c6f8cb588a 2025-11-30 15:40:21,803 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca88d481a0c6f8cb588a HTTP/1.1" 502 150 2025-11-30 15:40:21,803 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca88d481a0c6f8cb588a'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:48182 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:43962 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:40:37,803 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca88d481a0c6f8cb588a 2025-11-30 15:40:37,804 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca88d481a0c6f8cb588a HTTP/1.1" 502 150 2025-11-30 15:40:37,804 - lava_callback - ERROR - Error processing callback for node 692aca88d481a0c6f8cb588a: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca88d481a0c6f8cb588a (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca88d481a0c6f8cb588a (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca88d481a0c6f8cb588a (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:60200 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:53064 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:52374 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:46214 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:46216 - "GET /api/metrics HTTP/1.0" 200 OK 2025-11-30 15:41:16,350 - lava_callback - INFO - Received callback for node 6929c287498c72996de1f9ff INFO: 172.19.0.1:46228 - "POST /node/6929c287498c72996de1f9ff HTTP/1.0" 202 Accepted 2025-11-30 15:41:16,356 - lava_callback - INFO - Starting async job submit for node 6929c287498c72996de1f9ff 2025-11-30 15:41:16,356 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:41:16,367 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c287498c72996de1f9ff HTTP/1.1" 502 150 2025-11-30 15:41:16,367 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c287498c72996de1f9ff'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:41:16,367 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c287498c72996de1f9ff 2025-11-30 15:41:16,368 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c287498c72996de1f9ff HTTP/1.1" 502 150 2025-11-30 15:41:16,368 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c287498c72996de1f9ff'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:46244 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:41:17,102 - lava_callback - INFO - Received callback for node 6929c28b498c72996de1fa02 INFO: 172.19.0.1:46252 - "POST /node/6929c28b498c72996de1fa02 HTTP/1.0" 202 Accepted 2025-11-30 15:41:17,107 - lava_callback - INFO - Starting async job submit for node 6929c28b498c72996de1fa02 2025-11-30 15:41:17,108 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:41:17,118 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c28b498c72996de1fa02 HTTP/1.1" 502 150 2025-11-30 15:41:17,118 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c28b498c72996de1fa02'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:41:17,118 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c28b498c72996de1fa02 2025-11-30 15:41:17,119 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c28b498c72996de1fa02 HTTP/1.1" 502 150 2025-11-30 15:41:17,119 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c28b498c72996de1fa02'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:41:18,368 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c287498c72996de1f9ff 2025-11-30 15:41:18,369 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c287498c72996de1f9ff HTTP/1.1" 502 150 2025-11-30 15:41:18,369 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c287498c72996de1f9ff'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:41:19,119 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c28b498c72996de1fa02 2025-11-30 15:41:19,120 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c28b498c72996de1fa02 HTTP/1.1" 502 150 2025-11-30 15:41:19,120 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c28b498c72996de1fa02'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:41:22,370 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c287498c72996de1f9ff 2025-11-30 15:41:22,371 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c287498c72996de1f9ff HTTP/1.1" 502 150 2025-11-30 15:41:22,371 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c287498c72996de1f9ff'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:41:23,120 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c28b498c72996de1fa02 2025-11-30 15:41:23,122 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c28b498c72996de1fa02 HTTP/1.1" 502 150 2025-11-30 15:41:23,122 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c28b498c72996de1fa02'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:33438 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:41:30,371 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c287498c72996de1f9ff 2025-11-30 15:41:30,372 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c287498c72996de1f9ff HTTP/1.1" 502 150 2025-11-30 15:41:30,372 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c287498c72996de1f9ff'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:41:31,122 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c28b498c72996de1fa02 2025-11-30 15:41:31,123 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c28b498c72996de1fa02 HTTP/1.1" 502 150 2025-11-30 15:41:31,123 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c28b498c72996de1fa02'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:49952 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:41:46,373 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c287498c72996de1f9ff 2025-11-30 15:41:46,374 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c287498c72996de1f9ff HTTP/1.1" 502 150 2025-11-30 15:41:46,374 - lava_callback - ERROR - Error processing callback for node 6929c287498c72996de1f9ff: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c287498c72996de1f9ff (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c287498c72996de1f9ff (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c287498c72996de1f9ff (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:53706 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:41:47,124 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c28b498c72996de1fa02 2025-11-30 15:41:47,125 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c28b498c72996de1fa02 HTTP/1.1" 502 150 2025-11-30 15:41:47,125 - lava_callback - ERROR - Error processing callback for node 6929c28b498c72996de1fa02: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c28b498c72996de1fa02 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c28b498c72996de1fa02 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c28b498c72996de1fa02 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:56648 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:41222 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:53536 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:53546 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:53550 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:39028 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:59222 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:53426 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:41074 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:48794 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:51242 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:43:13,289 - lava_callback - INFO - Received callback for node 6929c2fc498c72996de1fa0b INFO: 172.19.0.1:51258 - "POST /node/6929c2fc498c72996de1fa0b HTTP/1.0" 202 Accepted 2025-11-30 15:43:13,294 - lava_callback - INFO - Starting async job submit for node 6929c2fc498c72996de1fa0b 2025-11-30 15:43:13,295 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:43:13,311 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c2fc498c72996de1fa0b HTTP/1.1" 502 150 2025-11-30 15:43:13,311 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c2fc498c72996de1fa0b'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:43:13,311 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c2fc498c72996de1fa0b 2025-11-30 15:43:13,312 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c2fc498c72996de1fa0b HTTP/1.1" 502 150 2025-11-30 15:43:13,312 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c2fc498c72996de1fa0b'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:51270 - "GET /api/metrics HTTP/1.0" 200 OK 2025-11-30 15:43:15,312 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c2fc498c72996de1fa0b 2025-11-30 15:43:15,313 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c2fc498c72996de1fa0b HTTP/1.1" 502 150 2025-11-30 15:43:15,313 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c2fc498c72996de1fa0b'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:51272 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:43:19,313 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c2fc498c72996de1fa0b 2025-11-30 15:43:19,318 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c2fc498c72996de1fa0b HTTP/1.1" 502 150 2025-11-30 15:43:19,318 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c2fc498c72996de1fa0b'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:33692 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:43:27,318 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c2fc498c72996de1fa0b 2025-11-30 15:43:27,320 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c2fc498c72996de1fa0b HTTP/1.1" 502 150 2025-11-30 15:43:27,320 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c2fc498c72996de1fa0b'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:43:31,695 - lava_callback - INFO - Received callback for node 692aca93d481a0c6f8cb589c INFO: 172.19.0.1:48134 - "POST /node/692aca93d481a0c6f8cb589c HTTP/1.0" 202 Accepted 2025-11-30 15:43:31,702 - lava_callback - INFO - Starting async job submit for node 692aca93d481a0c6f8cb589c 2025-11-30 15:43:31,727 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:43:31,740 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca93d481a0c6f8cb589c HTTP/1.1" 502 150 2025-11-30 15:43:31,740 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca93d481a0c6f8cb589c'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:43:31,741 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca93d481a0c6f8cb589c 2025-11-30 15:43:31,741 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca93d481a0c6f8cb589c HTTP/1.1" 502 150 2025-11-30 15:43:31,741 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca93d481a0c6f8cb589c'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:43:33,741 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca93d481a0c6f8cb589c 2025-11-30 15:43:33,743 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca93d481a0c6f8cb589c HTTP/1.1" 502 150 2025-11-30 15:43:33,743 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca93d481a0c6f8cb589c'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:48136 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:43:37,743 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca93d481a0c6f8cb589c 2025-11-30 15:43:37,745 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca93d481a0c6f8cb589c HTTP/1.1" 502 150 2025-11-30 15:43:37,745 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca93d481a0c6f8cb589c'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:43:43,320 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c2fc498c72996de1fa0b 2025-11-30 15:43:43,321 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c2fc498c72996de1fa0b HTTP/1.1" 502 150 2025-11-30 15:43:43,321 - lava_callback - ERROR - Error processing callback for node 6929c2fc498c72996de1fa0b: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c2fc498c72996de1fa0b (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c2fc498c72996de1fa0b (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c2fc498c72996de1fa0b (Caused by ResponseError('too many 502 error responses')) 2025-11-30 15:43:45,745 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca93d481a0c6f8cb589c 2025-11-30 15:43:45,746 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca93d481a0c6f8cb589c HTTP/1.1" 502 150 2025-11-30 15:43:45,746 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca93d481a0c6f8cb589c'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:38286 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:52120 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:44:01,746 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca93d481a0c6f8cb589c 2025-11-30 15:44:01,747 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca93d481a0c6f8cb589c HTTP/1.1" 502 150 2025-11-30 15:44:01,748 - lava_callback - ERROR - Error processing callback for node 692aca93d481a0c6f8cb589c: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca93d481a0c6f8cb589c (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca93d481a0c6f8cb589c (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca93d481a0c6f8cb589c (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:44852 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:36578 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:36582 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:36590 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:43546 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:46868 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:58512 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:44:48,604 - lava_callback - INFO - Received callback for node 692aca95d481a0c6f8cb589f INFO: 172.19.0.1:58516 - "POST /node/692aca95d481a0c6f8cb589f HTTP/1.0" 202 Accepted 2025-11-30 15:44:48,611 - lava_callback - INFO - Starting async job submit for node 692aca95d481a0c6f8cb589f 2025-11-30 15:44:48,712 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:44:48,751 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca95d481a0c6f8cb589f HTTP/1.1" 502 150 2025-11-30 15:44:48,751 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca95d481a0c6f8cb589f'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:44:48,751 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca95d481a0c6f8cb589f 2025-11-30 15:44:48,752 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca95d481a0c6f8cb589f HTTP/1.1" 502 150 2025-11-30 15:44:48,752 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca95d481a0c6f8cb589f'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:44:50,752 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca95d481a0c6f8cb589f 2025-11-30 15:44:50,753 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca95d481a0c6f8cb589f HTTP/1.1" 502 150 2025-11-30 15:44:50,753 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca95d481a0c6f8cb589f'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:44:54,753 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca95d481a0c6f8cb589f 2025-11-30 15:44:54,755 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca95d481a0c6f8cb589f HTTP/1.1" 502 150 2025-11-30 15:44:54,755 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca95d481a0c6f8cb589f'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:33294 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:45:02,755 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca95d481a0c6f8cb589f 2025-11-30 15:45:02,756 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca95d481a0c6f8cb589f HTTP/1.1" 502 150 2025-11-30 15:45:02,757 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca95d481a0c6f8cb589f'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:60226 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:41590 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:41600 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:41612 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:45:18,757 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca95d481a0c6f8cb589f 2025-11-30 15:45:18,758 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca95d481a0c6f8cb589f HTTP/1.1" 502 150 2025-11-30 15:45:18,758 - lava_callback - ERROR - Error processing callback for node 692aca95d481a0c6f8cb589f: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca95d481a0c6f8cb589f (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca95d481a0c6f8cb589f (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca95d481a0c6f8cb589f (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:50152 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:38966 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:35796 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:58794 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:32884 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:42686 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:42690 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:42694 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:55710 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:48668 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:36548 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:34948 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:56752 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:46100 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:46116 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:46132 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:60548 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:40380 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:47:46,292 - lava_callback - INFO - Received callback for node 6929c40e498c72996de1fa7c INFO: 172.19.0.1:35134 - "POST /node/6929c40e498c72996de1fa7c HTTP/1.0" 202 Accepted 2025-11-30 15:47:46,297 - lava_callback - INFO - Starting async job submit for node 6929c40e498c72996de1fa7c 2025-11-30 15:47:46,298 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:47:46,323 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40e498c72996de1fa7c HTTP/1.1" 502 150 2025-11-30 15:47:46,323 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c40e498c72996de1fa7c'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:46,324 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c40e498c72996de1fa7c 2025-11-30 15:47:46,324 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40e498c72996de1fa7c HTTP/1.1" 502 150 2025-11-30 15:47:46,324 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c40e498c72996de1fa7c'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:35140 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:47:46,954 - lava_callback - INFO - Received callback for node 6929c40d498c72996de1fa73 INFO: 172.19.0.1:35156 - "POST /node/6929c40d498c72996de1fa73 HTTP/1.0" 202 Accepted 2025-11-30 15:47:46,960 - lava_callback - INFO - Starting async job submit for node 6929c40d498c72996de1fa73 2025-11-30 15:47:46,961 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:47:46,979 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40d498c72996de1fa73 HTTP/1.1" 502 150 2025-11-30 15:47:46,979 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c40d498c72996de1fa73'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:46,979 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c40d498c72996de1fa73 2025-11-30 15:47:46,980 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40d498c72996de1fa73 HTTP/1.1" 502 150 2025-11-30 15:47:46,980 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c40d498c72996de1fa73'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:47,605 - lava_callback - INFO - Received callback for node 6929c40f498c72996de1fa88 2025-11-30 15:47:47,611 - lava_callback - INFO - Starting async job submit for node 6929c40f498c72996de1fa88 INFO: 172.19.0.1:35172 - "POST /node/6929c40f498c72996de1fa88 HTTP/1.0" 202 Accepted 2025-11-30 15:47:47,612 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:47:47,622 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40f498c72996de1fa88 HTTP/1.1" 502 150 2025-11-30 15:47:47,622 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c40f498c72996de1fa88'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:47,622 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c40f498c72996de1fa88 2025-11-30 15:47:47,623 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40f498c72996de1fa88 HTTP/1.1" 502 150 2025-11-30 15:47:47,623 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c40f498c72996de1fa88'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:48,277 - lava_callback - INFO - Received callback for node 6929c40e498c72996de1fa7f 2025-11-30 15:47:48,282 - lava_callback - INFO - Starting async job submit for node 6929c40e498c72996de1fa7f INFO: 172.19.0.1:35184 - "POST /node/6929c40e498c72996de1fa7f HTTP/1.0" 202 Accepted 2025-11-30 15:47:48,283 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:47:48,298 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40e498c72996de1fa7f HTTP/1.1" 502 150 2025-11-30 15:47:48,298 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c40e498c72996de1fa7f'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:48,299 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c40e498c72996de1fa7f 2025-11-30 15:47:48,299 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40e498c72996de1fa7f HTTP/1.1" 502 150 2025-11-30 15:47:48,299 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c40e498c72996de1fa7f'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:48,325 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c40e498c72996de1fa7c 2025-11-30 15:47:48,326 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40e498c72996de1fa7c HTTP/1.1" 502 150 2025-11-30 15:47:48,326 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c40e498c72996de1fa7c'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:48,936 - lava_callback - INFO - Received callback for node 6929c40f498c72996de1fa82 2025-11-30 15:47:48,941 - lava_callback - INFO - Starting async job submit for node 6929c40f498c72996de1fa82 2025-11-30 15:47:48,942 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 INFO: 172.19.0.1:35192 - "POST /node/6929c40f498c72996de1fa82 HTTP/1.0" 202 Accepted 2025-11-30 15:47:48,953 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40f498c72996de1fa82 HTTP/1.1" 502 150 2025-11-30 15:47:48,953 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c40f498c72996de1fa82'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:48,953 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c40f498c72996de1fa82 2025-11-30 15:47:48,954 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40f498c72996de1fa82 HTTP/1.1" 502 150 2025-11-30 15:47:48,954 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c40f498c72996de1fa82'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:48,980 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c40d498c72996de1fa73 2025-11-30 15:47:48,981 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40d498c72996de1fa73 HTTP/1.1" 502 150 2025-11-30 15:47:48,981 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c40d498c72996de1fa73'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:49,604 - lava_callback - INFO - Received callback for node 6929c410498c72996de1fa8b 2025-11-30 15:47:49,609 - lava_callback - INFO - Starting async job submit for node 6929c410498c72996de1fa8b INFO: 172.19.0.1:35204 - "POST /node/6929c410498c72996de1fa8b HTTP/1.0" 202 Accepted 2025-11-30 15:47:49,611 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:47:49,621 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c410498c72996de1fa8b HTTP/1.1" 502 150 2025-11-30 15:47:49,621 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c410498c72996de1fa8b'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:49,622 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c410498c72996de1fa8b 2025-11-30 15:47:49,622 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c410498c72996de1fa8b HTTP/1.1" 502 150 2025-11-30 15:47:49,622 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c410498c72996de1fa8b'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:49,623 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c40f498c72996de1fa88 2025-11-30 15:47:49,624 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40f498c72996de1fa88 HTTP/1.1" 502 150 2025-11-30 15:47:49,624 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c40f498c72996de1fa88'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:50,274 - lava_callback - INFO - Received callback for node 6929c411498c72996de1fa94 2025-11-30 15:47:50,279 - lava_callback - INFO - Starting async job submit for node 6929c411498c72996de1fa94 INFO: 172.19.0.1:36362 - "POST /node/6929c411498c72996de1fa94 HTTP/1.0" 202 Accepted 2025-11-30 15:47:50,280 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:47:50,292 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c411498c72996de1fa94 HTTP/1.1" 502 150 2025-11-30 15:47:50,292 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c411498c72996de1fa94'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:50,292 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c411498c72996de1fa94 2025-11-30 15:47:50,292 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c411498c72996de1fa94 HTTP/1.1" 502 150 2025-11-30 15:47:50,292 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c411498c72996de1fa94'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:50,299 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c40e498c72996de1fa7f 2025-11-30 15:47:50,300 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40e498c72996de1fa7f HTTP/1.1" 502 150 2025-11-30 15:47:50,300 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c40e498c72996de1fa7f'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:50,922 - lava_callback - INFO - Received callback for node 6929c411498c72996de1fa9a 2025-11-30 15:47:50,928 - lava_callback - INFO - Starting async job submit for node 6929c411498c72996de1fa9a 2025-11-30 15:47:50,928 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 INFO: 172.19.0.1:36366 - "POST /node/6929c411498c72996de1fa9a HTTP/1.0" 202 Accepted 2025-11-30 15:47:50,951 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c411498c72996de1fa9a HTTP/1.1" 502 150 2025-11-30 15:47:50,952 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c411498c72996de1fa9a'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:50,952 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c411498c72996de1fa9a 2025-11-30 15:47:50,952 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c411498c72996de1fa9a HTTP/1.1" 502 150 2025-11-30 15:47:50,952 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c411498c72996de1fa9a'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:50,954 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c40f498c72996de1fa82 2025-11-30 15:47:50,955 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40f498c72996de1fa82 HTTP/1.1" 502 150 2025-11-30 15:47:50,955 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c40f498c72996de1fa82'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:51,591 - lava_callback - INFO - Received callback for node 6929c412498c72996de1fa9d 2025-11-30 15:47:51,596 - lava_callback - INFO - Starting async job submit for node 6929c412498c72996de1fa9d INFO: 172.19.0.1:36368 - "POST /node/6929c412498c72996de1fa9d HTTP/1.0" 202 Accepted 2025-11-30 15:47:51,597 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:47:51,608 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c412498c72996de1fa9d HTTP/1.1" 502 150 2025-11-30 15:47:51,608 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c412498c72996de1fa9d'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:51,608 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c412498c72996de1fa9d 2025-11-30 15:47:51,609 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c412498c72996de1fa9d HTTP/1.1" 502 150 2025-11-30 15:47:51,609 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c412498c72996de1fa9d'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:51,622 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c410498c72996de1fa8b 2025-11-30 15:47:51,623 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c410498c72996de1fa8b HTTP/1.1" 502 150 2025-11-30 15:47:51,623 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c410498c72996de1fa8b'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:52,293 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c411498c72996de1fa94 2025-11-30 15:47:52,294 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c411498c72996de1fa94 HTTP/1.1" 502 150 2025-11-30 15:47:52,294 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c411498c72996de1fa94'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:52,326 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c40e498c72996de1fa7c 2025-11-30 15:47:52,327 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40e498c72996de1fa7c HTTP/1.1" 502 150 2025-11-30 15:47:52,327 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c40e498c72996de1fa7c'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:52,953 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c411498c72996de1fa9a 2025-11-30 15:47:52,954 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c411498c72996de1fa9a HTTP/1.1" 502 150 2025-11-30 15:47:52,954 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c411498c72996de1fa9a'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:52,981 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c40d498c72996de1fa73 2025-11-30 15:47:52,982 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40d498c72996de1fa73 HTTP/1.1" 502 150 2025-11-30 15:47:52,982 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c40d498c72996de1fa73'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:53,609 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c412498c72996de1fa9d 2025-11-30 15:47:53,610 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c412498c72996de1fa9d HTTP/1.1" 502 150 2025-11-30 15:47:53,611 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c412498c72996de1fa9d'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:53,624 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c40f498c72996de1fa88 2025-11-30 15:47:53,625 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40f498c72996de1fa88 HTTP/1.1" 502 150 2025-11-30 15:47:53,625 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c40f498c72996de1fa88'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:54,301 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c40e498c72996de1fa7f 2025-11-30 15:47:54,302 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40e498c72996de1fa7f HTTP/1.1" 502 150 2025-11-30 15:47:54,302 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c40e498c72996de1fa7f'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:54,955 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c40f498c72996de1fa82 2025-11-30 15:47:54,956 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40f498c72996de1fa82 HTTP/1.1" 502 150 2025-11-30 15:47:54,956 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c40f498c72996de1fa82'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:55,624 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c410498c72996de1fa8b 2025-11-30 15:47:55,625 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c410498c72996de1fa8b HTTP/1.1" 502 150 2025-11-30 15:47:55,625 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c410498c72996de1fa8b'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:56,294 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c411498c72996de1fa94 2025-11-30 15:47:56,295 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c411498c72996de1fa94 HTTP/1.1" 502 150 2025-11-30 15:47:56,295 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c411498c72996de1fa94'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:56,487 - lava_callback - INFO - Received callback for node 6929c41e498c72996de1faf4 2025-11-30 15:47:56,493 - lava_callback - INFO - Starting async job submit for node 6929c41e498c72996de1faf4 INFO: 172.19.0.1:36380 - "POST /node/6929c41e498c72996de1faf4 HTTP/1.0" 202 Accepted 2025-11-30 15:47:56,495 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:47:56,530 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c41e498c72996de1faf4 HTTP/1.1" 502 150 2025-11-30 15:47:56,530 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c41e498c72996de1faf4'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:56,530 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c41e498c72996de1faf4 2025-11-30 15:47:56,531 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c41e498c72996de1faf4 HTTP/1.1" 502 150 2025-11-30 15:47:56,531 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c41e498c72996de1faf4'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:36382 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:47:56,954 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c411498c72996de1fa9a 2025-11-30 15:47:56,955 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c411498c72996de1fa9a HTTP/1.1" 502 150 2025-11-30 15:47:56,955 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c411498c72996de1fa9a'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:57,123 - lava_callback - INFO - Received callback for node 6929c41f498c72996de1faf7 2025-11-30 15:47:57,147 - lava_callback - INFO - Starting async job submit for node 6929c41f498c72996de1faf7 INFO: 172.19.0.1:36398 - "POST /node/6929c41f498c72996de1faf7 HTTP/1.0" 202 Accepted 2025-11-30 15:47:57,148 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:47:57,158 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c41f498c72996de1faf7 HTTP/1.1" 502 150 2025-11-30 15:47:57,158 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c41f498c72996de1faf7'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:57,158 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c41f498c72996de1faf7 2025-11-30 15:47:57,159 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c41f498c72996de1faf7 HTTP/1.1" 502 150 2025-11-30 15:47:57,159 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c41f498c72996de1faf7'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:57,611 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c412498c72996de1fa9d 2025-11-30 15:47:57,612 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c412498c72996de1fa9d HTTP/1.1" 502 150 2025-11-30 15:47:57,612 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c412498c72996de1fa9d'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:58,532 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c41e498c72996de1faf4 2025-11-30 15:47:58,536 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c41e498c72996de1faf4 HTTP/1.1" 502 150 2025-11-30 15:47:58,536 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c41e498c72996de1faf4'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:47:59,159 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c41f498c72996de1faf7 2025-11-30 15:47:59,161 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c41f498c72996de1faf7 HTTP/1.1" 502 150 2025-11-30 15:47:59,161 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c41f498c72996de1faf7'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:48:00,327 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c40e498c72996de1fa7c 2025-11-30 15:48:00,328 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40e498c72996de1fa7c HTTP/1.1" 502 150 2025-11-30 15:48:00,328 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c40e498c72996de1fa7c'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:48:00,983 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c40d498c72996de1fa73 2025-11-30 15:48:00,987 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40d498c72996de1fa73 HTTP/1.1" 502 150 2025-11-30 15:48:00,987 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c40d498c72996de1fa73'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:48:01,625 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c40f498c72996de1fa88 2025-11-30 15:48:01,627 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40f498c72996de1fa88 HTTP/1.1" 502 150 2025-11-30 15:48:01,627 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c40f498c72996de1fa88'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:48:02,302 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c40e498c72996de1fa7f 2025-11-30 15:48:02,304 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40e498c72996de1fa7f HTTP/1.1" 502 150 2025-11-30 15:48:02,304 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c40e498c72996de1fa7f'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:48:02,536 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c41e498c72996de1faf4 2025-11-30 15:48:02,537 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c41e498c72996de1faf4 HTTP/1.1" 502 150 2025-11-30 15:48:02,537 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c41e498c72996de1faf4'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:48:02,956 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c40f498c72996de1fa82 2025-11-30 15:48:02,958 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40f498c72996de1fa82 HTTP/1.1" 502 150 2025-11-30 15:48:02,958 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c40f498c72996de1fa82'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:48:03,161 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c41f498c72996de1faf7 2025-11-30 15:48:03,162 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c41f498c72996de1faf7 HTTP/1.1" 502 150 2025-11-30 15:48:03,162 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c41f498c72996de1faf7'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:48:03,625 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c410498c72996de1fa8b 2025-11-30 15:48:03,627 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c410498c72996de1fa8b HTTP/1.1" 502 150 2025-11-30 15:48:03,627 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c410498c72996de1fa8b'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:48:04,295 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c411498c72996de1fa94 2025-11-30 15:48:04,297 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c411498c72996de1fa94 HTTP/1.1" 502 150 2025-11-30 15:48:04,297 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c411498c72996de1fa94'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:48:04,956 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c411498c72996de1fa9a 2025-11-30 15:48:04,957 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c411498c72996de1fa9a HTTP/1.1" 502 150 2025-11-30 15:48:04,957 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c411498c72996de1fa9a'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:48:05,612 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c412498c72996de1fa9d 2025-11-30 15:48:05,614 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c412498c72996de1fa9d HTTP/1.1" 502 150 2025-11-30 15:48:05,614 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c412498c72996de1fa9d'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:33714 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:48:10,537 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c41e498c72996de1faf4 2025-11-30 15:48:10,539 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c41e498c72996de1faf4 HTTP/1.1" 502 150 2025-11-30 15:48:10,539 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c41e498c72996de1faf4'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:48:11,163 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c41f498c72996de1faf7 2025-11-30 15:48:11,164 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c41f498c72996de1faf7 HTTP/1.1" 502 150 2025-11-30 15:48:11,165 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c41f498c72996de1faf7'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:34356 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:34370 - "GET /api/metrics HTTP/1.0" 200 OK 2025-11-30 15:48:16,329 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c40e498c72996de1fa7c 2025-11-30 15:48:16,330 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40e498c72996de1fa7c HTTP/1.1" 502 150 2025-11-30 15:48:16,330 - lava_callback - ERROR - Error processing callback for node 6929c40e498c72996de1fa7c: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c40e498c72996de1fa7c (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c40e498c72996de1fa7c (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c40e498c72996de1fa7c (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:34376 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:48:16,987 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c40d498c72996de1fa73 2025-11-30 15:48:16,989 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40d498c72996de1fa73 HTTP/1.1" 502 150 2025-11-30 15:48:16,989 - lava_callback - ERROR - Error processing callback for node 6929c40d498c72996de1fa73: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c40d498c72996de1fa73 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c40d498c72996de1fa73 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c40d498c72996de1fa73 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 15:48:17,627 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c40f498c72996de1fa88 2025-11-30 15:48:17,629 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40f498c72996de1fa88 HTTP/1.1" 502 150 2025-11-30 15:48:17,629 - lava_callback - ERROR - Error processing callback for node 6929c40f498c72996de1fa88: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c40f498c72996de1fa88 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c40f498c72996de1fa88 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c40f498c72996de1fa88 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 15:48:18,304 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c40e498c72996de1fa7f 2025-11-30 15:48:18,306 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40e498c72996de1fa7f HTTP/1.1" 502 150 2025-11-30 15:48:18,307 - lava_callback - ERROR - Error processing callback for node 6929c40e498c72996de1fa7f: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c40e498c72996de1fa7f (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c40e498c72996de1fa7f (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c40e498c72996de1fa7f (Caused by ResponseError('too many 502 error responses')) 2025-11-30 15:48:18,958 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c40f498c72996de1fa82 2025-11-30 15:48:18,964 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c40f498c72996de1fa82 HTTP/1.1" 502 150 2025-11-30 15:48:18,964 - lava_callback - ERROR - Error processing callback for node 6929c40f498c72996de1fa82: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c40f498c72996de1fa82 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c40f498c72996de1fa82 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c40f498c72996de1fa82 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 15:48:19,627 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c410498c72996de1fa8b 2025-11-30 15:48:19,628 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c410498c72996de1fa8b HTTP/1.1" 502 150 2025-11-30 15:48:19,628 - lava_callback - ERROR - Error processing callback for node 6929c410498c72996de1fa8b: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c410498c72996de1fa8b (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c410498c72996de1fa8b (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c410498c72996de1fa8b (Caused by ResponseError('too many 502 error responses')) 2025-11-30 15:48:20,297 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c411498c72996de1fa94 2025-11-30 15:48:20,298 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c411498c72996de1fa94 HTTP/1.1" 502 150 2025-11-30 15:48:20,299 - lava_callback - ERROR - Error processing callback for node 6929c411498c72996de1fa94: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c411498c72996de1fa94 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c411498c72996de1fa94 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c411498c72996de1fa94 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 15:48:20,957 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c411498c72996de1fa9a 2025-11-30 15:48:20,959 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c411498c72996de1fa9a HTTP/1.1" 502 150 2025-11-30 15:48:20,959 - lava_callback - ERROR - Error processing callback for node 6929c411498c72996de1fa9a: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c411498c72996de1fa9a (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c411498c72996de1fa9a (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c411498c72996de1fa9a (Caused by ResponseError('too many 502 error responses')) 2025-11-30 15:48:21,614 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c412498c72996de1fa9d 2025-11-30 15:48:21,615 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c412498c72996de1fa9d HTTP/1.1" 502 150 2025-11-30 15:48:21,616 - lava_callback - ERROR - Error processing callback for node 6929c412498c72996de1fa9d: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c412498c72996de1fa9d (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c412498c72996de1fa9d (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c412498c72996de1fa9d (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:38276 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:48:26,539 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c41e498c72996de1faf4 2025-11-30 15:48:26,540 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c41e498c72996de1faf4 HTTP/1.1" 502 150 2025-11-30 15:48:26,540 - lava_callback - ERROR - Error processing callback for node 6929c41e498c72996de1faf4: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c41e498c72996de1faf4 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c41e498c72996de1faf4 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c41e498c72996de1faf4 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 15:48:27,165 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c41f498c72996de1faf7 2025-11-30 15:48:27,166 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c41f498c72996de1faf7 HTTP/1.1" 502 150 2025-11-30 15:48:27,166 - lava_callback - ERROR - Error processing callback for node 6929c41f498c72996de1faf7: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c41f498c72996de1faf7 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c41f498c72996de1faf7 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c41f498c72996de1faf7 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:51446 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:50020 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:41072 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:57530 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:58984 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:58992 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:58996 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:54518 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:49:32,400 - lava_callback - INFO - Received callback for node 692aca96d481a0c6f8cb58a2 INFO: 172.19.0.1:35260 - "POST /node/692aca96d481a0c6f8cb58a2 HTTP/1.0" 202 Accepted 2025-11-30 15:49:32,413 - lava_callback - INFO - Starting async job submit for node 692aca96d481a0c6f8cb58a2 2025-11-30 15:49:32,868 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:49:32,888 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca96d481a0c6f8cb58a2 HTTP/1.1" 502 150 2025-11-30 15:49:32,888 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca96d481a0c6f8cb58a2'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:49:32,890 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca96d481a0c6f8cb58a2 2025-11-30 15:49:32,894 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca96d481a0c6f8cb58a2 HTTP/1.1" 502 150 2025-11-30 15:49:32,894 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca96d481a0c6f8cb58a2'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:49:34,895 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca96d481a0c6f8cb58a2 2025-11-30 15:49:34,896 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca96d481a0c6f8cb58a2 HTTP/1.1" 502 150 2025-11-30 15:49:34,896 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca96d481a0c6f8cb58a2'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:35270 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:49:38,896 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca96d481a0c6f8cb58a2 2025-11-30 15:49:38,897 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca96d481a0c6f8cb58a2 HTTP/1.1" 502 150 2025-11-30 15:49:38,897 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca96d481a0c6f8cb58a2'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:40984 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:49:46,898 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca96d481a0c6f8cb58a2 2025-11-30 15:49:46,899 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca96d481a0c6f8cb58a2 HTTP/1.1" 502 150 2025-11-30 15:49:46,899 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca96d481a0c6f8cb58a2'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:56488 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:50:02,899 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca96d481a0c6f8cb58a2 2025-11-30 15:50:02,901 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca96d481a0c6f8cb58a2 HTTP/1.1" 502 150 2025-11-30 15:50:02,901 - lava_callback - ERROR - Error processing callback for node 692aca96d481a0c6f8cb58a2: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca96d481a0c6f8cb58a2 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca96d481a0c6f8cb58a2 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca96d481a0c6f8cb58a2 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:48540 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:50:12,029 - lava_callback - INFO - Received callback for node 692aca9ad481a0c6f8cb58a8 INFO: 172.19.0.1:58660 - "POST /node/692aca9ad481a0c6f8cb58a8 HTTP/1.0" 202 Accepted 2025-11-30 15:50:12,049 - lava_callback - INFO - Starting async job submit for node 692aca9ad481a0c6f8cb58a8 INFO: 172.19.0.1:58666 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:50:12,333 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:50:12,346 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca9ad481a0c6f8cb58a8 HTTP/1.1" 502 150 2025-11-30 15:50:12,346 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca9ad481a0c6f8cb58a8'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:50:12,347 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca9ad481a0c6f8cb58a8 2025-11-30 15:50:12,348 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca9ad481a0c6f8cb58a8 HTTP/1.1" 502 150 2025-11-30 15:50:12,348 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca9ad481a0c6f8cb58a8'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:50:14,348 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca9ad481a0c6f8cb58a8 2025-11-30 15:50:14,349 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca9ad481a0c6f8cb58a8 HTTP/1.1" 502 150 2025-11-30 15:50:14,349 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca9ad481a0c6f8cb58a8'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:58668 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:58674 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:50:18,350 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca9ad481a0c6f8cb58a8 2025-11-30 15:50:18,351 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca9ad481a0c6f8cb58a8 HTTP/1.1" 502 150 2025-11-30 15:50:18,351 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca9ad481a0c6f8cb58a8'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:50:26,351 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca9ad481a0c6f8cb58a8 2025-11-30 15:50:26,352 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca9ad481a0c6f8cb58a8 HTTP/1.1" 502 150 2025-11-30 15:50:26,353 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca9ad481a0c6f8cb58a8'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:44182 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:41846 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:50:42,353 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca9ad481a0c6f8cb58a8 2025-11-30 15:50:42,355 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca9ad481a0c6f8cb58a8 HTTP/1.1" 502 150 2025-11-30 15:50:42,355 - lava_callback - ERROR - Error processing callback for node 692aca9ad481a0c6f8cb58a8: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca9ad481a0c6f8cb58a8 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca9ad481a0c6f8cb58a8 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca9ad481a0c6f8cb58a8 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:57370 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:35740 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:40570 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:48474 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:48482 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:48484 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:39116 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:34812 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:55770 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:57766 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:56974 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:52:08,468 - lava_callback - INFO - Received callback for node 692aca9bd481a0c6f8cb58ab INFO: 172.19.0.1:56980 - "POST /node/692aca9bd481a0c6f8cb58ab HTTP/1.0" 202 Accepted 2025-11-30 15:52:08,475 - lava_callback - INFO - Starting async job submit for node 692aca9bd481a0c6f8cb58ab 2025-11-30 15:52:08,516 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:52:08,525 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca9bd481a0c6f8cb58ab HTTP/1.1" 502 150 2025-11-30 15:52:08,526 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca9bd481a0c6f8cb58ab'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:08,526 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca9bd481a0c6f8cb58ab 2025-11-30 15:52:08,526 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca9bd481a0c6f8cb58ab HTTP/1.1" 502 150 2025-11-30 15:52:08,526 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca9bd481a0c6f8cb58ab'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:10,527 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca9bd481a0c6f8cb58ab 2025-11-30 15:52:10,528 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca9bd481a0c6f8cb58ab HTTP/1.1" 502 150 2025-11-30 15:52:10,528 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca9bd481a0c6f8cb58ab'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:12,121 - lava_callback - INFO - Received callback for node 692aca9dd481a0c6f8cb58ae INFO: 172.19.0.1:35594 - "POST /node/692aca9dd481a0c6f8cb58ae HTTP/1.0" 202 Accepted 2025-11-30 15:52:12,132 - lava_callback - INFO - Starting async job submit for node 692aca9dd481a0c6f8cb58ae 2025-11-30 15:52:12,192 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:52:12,229 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca9dd481a0c6f8cb58ae HTTP/1.1" 502 150 2025-11-30 15:52:12,229 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca9dd481a0c6f8cb58ae'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:12,230 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca9dd481a0c6f8cb58ae 2025-11-30 15:52:12,231 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca9dd481a0c6f8cb58ae HTTP/1.1" 502 150 2025-11-30 15:52:12,231 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca9dd481a0c6f8cb58ae'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:35610 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:52:14,231 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca9dd481a0c6f8cb58ae 2025-11-30 15:52:14,234 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca9dd481a0c6f8cb58ae HTTP/1.1" 502 150 2025-11-30 15:52:14,234 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca9dd481a0c6f8cb58ae'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:14,528 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca9bd481a0c6f8cb58ab 2025-11-30 15:52:14,529 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca9bd481a0c6f8cb58ab HTTP/1.1" 502 150 2025-11-30 15:52:14,530 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca9bd481a0c6f8cb58ab'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:35612 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:35628 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:52:18,234 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca9dd481a0c6f8cb58ae 2025-11-30 15:52:18,235 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca9dd481a0c6f8cb58ae HTTP/1.1" 502 150 2025-11-30 15:52:18,235 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca9dd481a0c6f8cb58ae'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:22,530 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca9bd481a0c6f8cb58ab 2025-11-30 15:52:22,531 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca9bd481a0c6f8cb58ab HTTP/1.1" 502 150 2025-11-30 15:52:22,531 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca9bd481a0c6f8cb58ab'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:26,236 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca9dd481a0c6f8cb58ae 2025-11-30 15:52:26,237 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca9dd481a0c6f8cb58ae HTTP/1.1" 502 150 2025-11-30 15:52:26,237 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca9dd481a0c6f8cb58ae'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:41786 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:52:26,539 - lava_callback - INFO - Received callback for node 6929c525498c72996de1fbee INFO: 172.19.0.1:41794 - "POST /node/6929c525498c72996de1fbee HTTP/1.0" 202 Accepted 2025-11-30 15:52:26,544 - lava_callback - INFO - Starting async job submit for node 6929c525498c72996de1fbee 2025-11-30 15:52:26,545 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:52:26,555 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c525498c72996de1fbee HTTP/1.1" 502 150 2025-11-30 15:52:26,555 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c525498c72996de1fbee'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:26,555 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c525498c72996de1fbee 2025-11-30 15:52:26,556 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c525498c72996de1fbee HTTP/1.1" 502 150 2025-11-30 15:52:26,556 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c525498c72996de1fbee'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:27,209 - lava_callback - INFO - Received callback for node 6929c527498c72996de1fbf5 INFO: 172.19.0.1:41798 - "POST /node/6929c527498c72996de1fbf5 HTTP/1.0" 202 Accepted 2025-11-30 15:52:27,214 - lava_callback - INFO - Starting async job submit for node 6929c527498c72996de1fbf5 2025-11-30 15:52:27,215 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:52:27,225 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c527498c72996de1fbf5 HTTP/1.1" 502 150 2025-11-30 15:52:27,225 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c527498c72996de1fbf5'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:27,225 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c527498c72996de1fbf5 2025-11-30 15:52:27,226 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c527498c72996de1fbf5 HTTP/1.1" 502 150 2025-11-30 15:52:27,226 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c527498c72996de1fbf5'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:27,879 - lava_callback - INFO - Received callback for node 6929c529498c72996de1fbf8 INFO: 172.19.0.1:41804 - "POST /node/6929c529498c72996de1fbf8 HTTP/1.0" 202 Accepted 2025-11-30 15:52:27,884 - lava_callback - INFO - Starting async job submit for node 6929c529498c72996de1fbf8 2025-11-30 15:52:27,885 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:52:27,898 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c529498c72996de1fbf8 HTTP/1.1" 502 150 2025-11-30 15:52:27,898 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c529498c72996de1fbf8'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:27,898 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c529498c72996de1fbf8 2025-11-30 15:52:27,899 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c529498c72996de1fbf8 HTTP/1.1" 502 150 2025-11-30 15:52:27,899 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c529498c72996de1fbf8'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:28,556 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c525498c72996de1fbee 2025-11-30 15:52:28,557 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c525498c72996de1fbee HTTP/1.1" 502 150 2025-11-30 15:52:28,557 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c525498c72996de1fbee'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:28,559 - lava_callback - INFO - Received callback for node 6929c52a498c72996de1fbfb INFO: 172.19.0.1:41812 - "POST /node/6929c52a498c72996de1fbfb HTTP/1.0" 202 Accepted 2025-11-30 15:52:28,565 - lava_callback - INFO - Starting async job submit for node 6929c52a498c72996de1fbfb 2025-11-30 15:52:28,565 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:52:28,575 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c52a498c72996de1fbfb HTTP/1.1" 502 150 2025-11-30 15:52:28,575 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c52a498c72996de1fbfb'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:28,575 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c52a498c72996de1fbfb 2025-11-30 15:52:28,576 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c52a498c72996de1fbfb HTTP/1.1" 502 150 2025-11-30 15:52:28,576 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c52a498c72996de1fbfb'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:29,226 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c527498c72996de1fbf5 2025-11-30 15:52:29,227 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c527498c72996de1fbf5 HTTP/1.1" 502 150 2025-11-30 15:52:29,227 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c527498c72996de1fbf5'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:29,239 - lava_callback - INFO - Received callback for node 6929c52c498c72996de1fbfe INFO: 172.19.0.1:41822 - "POST /node/6929c52c498c72996de1fbfe HTTP/1.0" 202 Accepted 2025-11-30 15:52:29,245 - lava_callback - INFO - Starting async job submit for node 6929c52c498c72996de1fbfe 2025-11-30 15:52:29,246 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:52:29,263 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c52c498c72996de1fbfe HTTP/1.1" 502 150 2025-11-30 15:52:29,263 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c52c498c72996de1fbfe'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:29,263 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c52c498c72996de1fbfe 2025-11-30 15:52:29,264 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c52c498c72996de1fbfe HTTP/1.1" 502 150 2025-11-30 15:52:29,264 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c52c498c72996de1fbfe'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:29,899 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c529498c72996de1fbf8 2025-11-30 15:52:29,900 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c529498c72996de1fbf8 HTTP/1.1" 502 150 2025-11-30 15:52:29,900 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c529498c72996de1fbf8'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:29,923 - lava_callback - INFO - Received callback for node 6929c52e498c72996de1fc01 INFO: 172.19.0.1:39270 - "POST /node/6929c52e498c72996de1fc01 HTTP/1.0" 202 Accepted 2025-11-30 15:52:29,933 - lava_callback - INFO - Starting async job submit for node 6929c52e498c72996de1fc01 2025-11-30 15:52:29,934 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:52:29,946 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c52e498c72996de1fc01 HTTP/1.1" 502 150 2025-11-30 15:52:29,947 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c52e498c72996de1fc01'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:29,947 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c52e498c72996de1fc01 2025-11-30 15:52:29,948 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c52e498c72996de1fc01 HTTP/1.1" 502 150 2025-11-30 15:52:29,948 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c52e498c72996de1fc01'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:30,576 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c52a498c72996de1fbfb 2025-11-30 15:52:30,578 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c52a498c72996de1fbfb HTTP/1.1" 502 150 2025-11-30 15:52:30,578 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c52a498c72996de1fbfb'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:31,264 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c52c498c72996de1fbfe 2025-11-30 15:52:31,265 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c52c498c72996de1fbfe HTTP/1.1" 502 150 2025-11-30 15:52:31,265 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c52c498c72996de1fbfe'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:31,948 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c52e498c72996de1fc01 2025-11-30 15:52:31,949 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c52e498c72996de1fc01 HTTP/1.1" 502 150 2025-11-30 15:52:31,949 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c52e498c72996de1fc01'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:32,558 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c525498c72996de1fbee 2025-11-30 15:52:32,559 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c525498c72996de1fbee HTTP/1.1" 502 150 2025-11-30 15:52:32,559 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c525498c72996de1fbee'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:33,228 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c527498c72996de1fbf5 2025-11-30 15:52:33,229 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c527498c72996de1fbf5 HTTP/1.1" 502 150 2025-11-30 15:52:33,229 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c527498c72996de1fbf5'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:33,900 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c529498c72996de1fbf8 2025-11-30 15:52:33,901 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c529498c72996de1fbf8 HTTP/1.1" 502 150 2025-11-30 15:52:33,902 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c529498c72996de1fbf8'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:34,578 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c52a498c72996de1fbfb 2025-11-30 15:52:34,582 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c52a498c72996de1fbfb HTTP/1.1" 502 150 2025-11-30 15:52:34,582 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c52a498c72996de1fbfb'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:35,265 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c52c498c72996de1fbfe 2025-11-30 15:52:35,267 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c52c498c72996de1fbfe HTTP/1.1" 502 150 2025-11-30 15:52:35,267 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c52c498c72996de1fbfe'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:35,949 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c52e498c72996de1fc01 2025-11-30 15:52:35,951 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c52e498c72996de1fc01 HTTP/1.1" 502 150 2025-11-30 15:52:35,951 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c52e498c72996de1fc01'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:39286 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:52:38,531 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca9bd481a0c6f8cb58ab 2025-11-30 15:52:38,533 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca9bd481a0c6f8cb58ab HTTP/1.1" 502 150 2025-11-30 15:52:38,533 - lava_callback - ERROR - Error processing callback for node 692aca9bd481a0c6f8cb58ab: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca9bd481a0c6f8cb58ab (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca9bd481a0c6f8cb58ab (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca9bd481a0c6f8cb58ab (Caused by ResponseError('too many 502 error responses')) 2025-11-30 15:52:40,559 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c525498c72996de1fbee 2025-11-30 15:52:40,561 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c525498c72996de1fbee HTTP/1.1" 502 150 2025-11-30 15:52:40,561 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c525498c72996de1fbee'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:41,229 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c527498c72996de1fbf5 2025-11-30 15:52:41,231 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c527498c72996de1fbf5 HTTP/1.1" 502 150 2025-11-30 15:52:41,231 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c527498c72996de1fbf5'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:41,902 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c529498c72996de1fbf8 2025-11-30 15:52:41,903 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c529498c72996de1fbf8 HTTP/1.1" 502 150 2025-11-30 15:52:41,903 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c529498c72996de1fbf8'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:42,237 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca9dd481a0c6f8cb58ae 2025-11-30 15:52:42,239 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca9dd481a0c6f8cb58ae HTTP/1.1" 502 150 2025-11-30 15:52:42,239 - lava_callback - ERROR - Error processing callback for node 692aca9dd481a0c6f8cb58ae: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca9dd481a0c6f8cb58ae (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca9dd481a0c6f8cb58ae (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca9dd481a0c6f8cb58ae (Caused by ResponseError('too many 502 error responses')) 2025-11-30 15:52:42,582 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c52a498c72996de1fbfb 2025-11-30 15:52:42,583 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c52a498c72996de1fbfb HTTP/1.1" 502 150 2025-11-30 15:52:42,584 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c52a498c72996de1fbfb'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:43,267 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c52c498c72996de1fbfe 2025-11-30 15:52:43,269 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c52c498c72996de1fbfe HTTP/1.1" 502 150 2025-11-30 15:52:43,269 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c52c498c72996de1fbfe'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:43,517 - lava_callback - INFO - Received callback for node 6929c53a498c72996de1fc16 INFO: 172.19.0.1:35758 - "POST /node/6929c53a498c72996de1fc16 HTTP/1.0" 202 Accepted 2025-11-30 15:52:43,526 - lava_callback - INFO - Starting async job submit for node 6929c53a498c72996de1fc16 2025-11-30 15:52:43,528 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:52:43,549 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c53a498c72996de1fc16 HTTP/1.1" 502 150 2025-11-30 15:52:43,549 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c53a498c72996de1fc16'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:43,549 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c53a498c72996de1fc16 2025-11-30 15:52:43,550 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c53a498c72996de1fc16 HTTP/1.1" 502 150 2025-11-30 15:52:43,550 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c53a498c72996de1fc16'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:43,951 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c52e498c72996de1fc01 2025-11-30 15:52:43,952 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c52e498c72996de1fc01 HTTP/1.1" 502 150 2025-11-30 15:52:43,952 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c52e498c72996de1fc01'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:44,198 - lava_callback - INFO - Received callback for node 6929c53d498c72996de1fc19 INFO: 172.19.0.1:35774 - "POST /node/6929c53d498c72996de1fc19 HTTP/1.0" 202 Accepted 2025-11-30 15:52:44,204 - lava_callback - INFO - Starting async job submit for node 6929c53d498c72996de1fc19 2025-11-30 15:52:44,205 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:52:44,239 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c53d498c72996de1fc19 HTTP/1.1" 502 150 2025-11-30 15:52:44,239 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c53d498c72996de1fc19'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:44,239 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c53d498c72996de1fc19 2025-11-30 15:52:44,240 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c53d498c72996de1fc19 HTTP/1.1" 502 150 2025-11-30 15:52:44,240 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c53d498c72996de1fc19'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:45,550 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c53a498c72996de1fc16 2025-11-30 15:52:45,552 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c53a498c72996de1fc16 HTTP/1.1" 502 150 2025-11-30 15:52:45,552 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c53a498c72996de1fc16'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:46,240 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c53d498c72996de1fc19 2025-11-30 15:52:46,241 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c53d498c72996de1fc19 HTTP/1.1" 502 150 2025-11-30 15:52:46,241 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c53d498c72996de1fc19'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:35786 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:52:49,552 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c53a498c72996de1fc16 2025-11-30 15:52:49,553 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c53a498c72996de1fc16 HTTP/1.1" 502 150 2025-11-30 15:52:49,553 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c53a498c72996de1fc16'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:50,241 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c53d498c72996de1fc19 2025-11-30 15:52:50,242 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c53d498c72996de1fc19 HTTP/1.1" 502 150 2025-11-30 15:52:50,243 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c53d498c72996de1fc19'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:39094 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:52:56,561 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c525498c72996de1fbee 2025-11-30 15:52:56,562 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c525498c72996de1fbee HTTP/1.1" 502 150 2025-11-30 15:52:56,562 - lava_callback - ERROR - Error processing callback for node 6929c525498c72996de1fbee: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c525498c72996de1fbee (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c525498c72996de1fbee (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c525498c72996de1fbee (Caused by ResponseError('too many 502 error responses')) 2025-11-30 15:52:57,231 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c527498c72996de1fbf5 2025-11-30 15:52:57,233 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c527498c72996de1fbf5 HTTP/1.1" 502 150 2025-11-30 15:52:57,233 - lava_callback - ERROR - Error processing callback for node 6929c527498c72996de1fbf5: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c527498c72996de1fbf5 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c527498c72996de1fbf5 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c527498c72996de1fbf5 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 15:52:57,553 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c53a498c72996de1fc16 2025-11-30 15:52:57,554 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c53a498c72996de1fc16 HTTP/1.1" 502 150 2025-11-30 15:52:57,555 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c53a498c72996de1fc16'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:57,903 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c529498c72996de1fbf8 2025-11-30 15:52:57,904 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c529498c72996de1fbf8 HTTP/1.1" 502 150 2025-11-30 15:52:57,904 - lava_callback - ERROR - Error processing callback for node 6929c529498c72996de1fbf8: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c529498c72996de1fbf8 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c529498c72996de1fbf8 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c529498c72996de1fbf8 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 15:52:58,243 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c53d498c72996de1fc19 2025-11-30 15:52:58,244 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c53d498c72996de1fc19 HTTP/1.1" 502 150 2025-11-30 15:52:58,244 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c53d498c72996de1fc19'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:52:58,584 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c52a498c72996de1fbfb 2025-11-30 15:52:58,588 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c52a498c72996de1fbfb HTTP/1.1" 502 150 2025-11-30 15:52:58,588 - lava_callback - ERROR - Error processing callback for node 6929c52a498c72996de1fbfb: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c52a498c72996de1fbfb (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c52a498c72996de1fbfb (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c52a498c72996de1fbfb (Caused by ResponseError('too many 502 error responses')) 2025-11-30 15:52:59,269 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c52c498c72996de1fbfe 2025-11-30 15:52:59,270 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c52c498c72996de1fbfe HTTP/1.1" 502 150 2025-11-30 15:52:59,271 - lava_callback - ERROR - Error processing callback for node 6929c52c498c72996de1fbfe: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c52c498c72996de1fbfe (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c52c498c72996de1fbfe (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c52c498c72996de1fbfe (Caused by ResponseError('too many 502 error responses')) 2025-11-30 15:52:59,952 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c52e498c72996de1fc01 2025-11-30 15:52:59,954 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c52e498c72996de1fc01 HTTP/1.1" 502 150 2025-11-30 15:52:59,954 - lava_callback - ERROR - Error processing callback for node 6929c52e498c72996de1fc01: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c52e498c72996de1fc01 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c52e498c72996de1fc01 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c52e498c72996de1fc01 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:33738 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:57362 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:53:13,555 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c53a498c72996de1fc16 2025-11-30 15:53:13,556 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c53a498c72996de1fc16 HTTP/1.1" 502 150 2025-11-30 15:53:13,556 - lava_callback - ERROR - Error processing callback for node 6929c53a498c72996de1fc16: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c53a498c72996de1fc16 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c53a498c72996de1fc16 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c53a498c72996de1fc16 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 15:53:14,244 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c53d498c72996de1fc19 2025-11-30 15:53:14,246 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c53d498c72996de1fc19 HTTP/1.1" 502 150 2025-11-30 15:53:14,246 - lava_callback - ERROR - Error processing callback for node 6929c53d498c72996de1fc19: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c53d498c72996de1fc19 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c53d498c72996de1fc19 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c53d498c72996de1fc19 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:57368 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:57384 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:39968 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:53:32,132 - lava_callback - INFO - Received callback for node 692aca9fd481a0c6f8cb58b1 INFO: 172.19.0.1:50024 - "POST /node/692aca9fd481a0c6f8cb58b1 HTTP/1.0" 202 Accepted 2025-11-30 15:53:32,138 - lava_callback - INFO - Starting async job submit for node 692aca9fd481a0c6f8cb58b1 2025-11-30 15:53:32,163 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:53:32,203 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca9fd481a0c6f8cb58b1 HTTP/1.1" 502 150 2025-11-30 15:53:32,203 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca9fd481a0c6f8cb58b1'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:53:32,204 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca9fd481a0c6f8cb58b1 2025-11-30 15:53:32,204 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca9fd481a0c6f8cb58b1 HTTP/1.1" 502 150 2025-11-30 15:53:32,204 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca9fd481a0c6f8cb58b1'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:53:34,204 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca9fd481a0c6f8cb58b1 2025-11-30 15:53:34,206 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca9fd481a0c6f8cb58b1 HTTP/1.1" 502 150 2025-11-30 15:53:34,206 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca9fd481a0c6f8cb58b1'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:50032 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:53:38,206 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca9fd481a0c6f8cb58b1 2025-11-30 15:53:38,207 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca9fd481a0c6f8cb58b1 HTTP/1.1" 502 150 2025-11-30 15:53:38,208 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca9fd481a0c6f8cb58b1'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:53:46,208 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca9fd481a0c6f8cb58b1 2025-11-30 15:53:46,209 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca9fd481a0c6f8cb58b1 HTTP/1.1" 502 150 2025-11-30 15:53:46,209 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca9fd481a0c6f8cb58b1'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:47736 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:53:46,600 - lava_callback - INFO - Received callback for node 6929c580498c72996de1fc65 INFO: 172.19.0.1:47752 - "POST /node/6929c580498c72996de1fc65 HTTP/1.0" 202 Accepted 2025-11-30 15:53:46,611 - lava_callback - INFO - Starting async job submit for node 6929c580498c72996de1fc65 2025-11-30 15:53:46,616 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:53:46,670 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c580498c72996de1fc65 HTTP/1.1" 502 150 2025-11-30 15:53:46,670 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c580498c72996de1fc65'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:53:46,670 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c580498c72996de1fc65 2025-11-30 15:53:46,671 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c580498c72996de1fc65 HTTP/1.1" 502 150 2025-11-30 15:53:46,671 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c580498c72996de1fc65'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:53:47,289 - lava_callback - INFO - Received callback for node 6929c583498c72996de1fc68 INFO: 172.19.0.1:47756 - "POST /node/6929c583498c72996de1fc68 HTTP/1.0" 202 Accepted 2025-11-30 15:53:47,295 - lava_callback - INFO - Starting async job submit for node 6929c583498c72996de1fc68 2025-11-30 15:53:47,296 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:53:47,306 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c583498c72996de1fc68 HTTP/1.1" 502 150 2025-11-30 15:53:47,306 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c583498c72996de1fc68'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:53:47,306 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c583498c72996de1fc68 2025-11-30 15:53:47,307 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c583498c72996de1fc68 HTTP/1.1" 502 150 2025-11-30 15:53:47,307 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c583498c72996de1fc68'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:53:47,949 - lava_callback - INFO - Received callback for node 6929c584498c72996de1fc6b INFO: 172.19.0.1:47760 - "POST /node/6929c584498c72996de1fc6b HTTP/1.0" 202 Accepted 2025-11-30 15:53:47,955 - lava_callback - INFO - Starting async job submit for node 6929c584498c72996de1fc6b 2025-11-30 15:53:47,955 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:53:47,988 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c584498c72996de1fc6b HTTP/1.1" 502 150 2025-11-30 15:53:47,988 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c584498c72996de1fc6b'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:53:47,988 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c584498c72996de1fc6b 2025-11-30 15:53:47,991 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c584498c72996de1fc6b HTTP/1.1" 502 150 2025-11-30 15:53:47,991 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c584498c72996de1fc6b'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:53:48,626 - lava_callback - INFO - Received callback for node 6929c586498c72996de1fc6e INFO: 172.19.0.1:47774 - "POST /node/6929c586498c72996de1fc6e HTTP/1.0" 202 Accepted 2025-11-30 15:53:48,631 - lava_callback - INFO - Starting async job submit for node 6929c586498c72996de1fc6e 2025-11-30 15:53:48,632 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:53:48,670 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c586498c72996de1fc6e HTTP/1.1" 502 150 2025-11-30 15:53:48,670 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c586498c72996de1fc6e'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:53:48,670 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c586498c72996de1fc6e 2025-11-30 15:53:48,671 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c586498c72996de1fc6e HTTP/1.1" 502 150 2025-11-30 15:53:48,671 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c586498c72996de1fc6e'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:53:48,671 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c580498c72996de1fc65 2025-11-30 15:53:48,672 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c580498c72996de1fc65 HTTP/1.1" 502 150 2025-11-30 15:53:48,672 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c580498c72996de1fc65'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:53:49,298 - lava_callback - INFO - Received callback for node 6929c588498c72996de1fc71 INFO: 172.19.0.1:47782 - "POST /node/6929c588498c72996de1fc71 HTTP/1.0" 202 Accepted 2025-11-30 15:53:49,304 - lava_callback - INFO - Starting async job submit for node 6929c588498c72996de1fc71 2025-11-30 15:53:49,305 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:53:49,307 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c583498c72996de1fc68 2025-11-30 15:53:49,308 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c583498c72996de1fc68 HTTP/1.1" 502 150 2025-11-30 15:53:49,308 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c583498c72996de1fc68'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:53:49,334 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c588498c72996de1fc71 HTTP/1.1" 502 150 2025-11-30 15:53:49,334 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c588498c72996de1fc71'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:53:49,334 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c588498c72996de1fc71 2025-11-30 15:53:49,335 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c588498c72996de1fc71 HTTP/1.1" 502 150 2025-11-30 15:53:49,335 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c588498c72996de1fc71'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:53:49,991 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c584498c72996de1fc6b 2025-11-30 15:53:49,992 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c584498c72996de1fc6b HTTP/1.1" 502 150 2025-11-30 15:53:49,992 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c584498c72996de1fc6b'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:53:50,671 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c586498c72996de1fc6e 2025-11-30 15:53:50,672 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c586498c72996de1fc6e HTTP/1.1" 502 150 2025-11-30 15:53:50,672 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c586498c72996de1fc6e'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:53:51,335 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c588498c72996de1fc71 2025-11-30 15:53:51,336 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c588498c72996de1fc71 HTTP/1.1" 502 150 2025-11-30 15:53:51,337 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c588498c72996de1fc71'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:53:52,672 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c580498c72996de1fc65 2025-11-30 15:53:52,674 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c580498c72996de1fc65 HTTP/1.1" 502 150 2025-11-30 15:53:52,674 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c580498c72996de1fc65'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:53:53,309 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c583498c72996de1fc68 2025-11-30 15:53:53,310 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c583498c72996de1fc68 HTTP/1.1" 502 150 2025-11-30 15:53:53,310 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c583498c72996de1fc68'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:53:53,992 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c584498c72996de1fc6b 2025-11-30 15:53:53,993 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c584498c72996de1fc6b HTTP/1.1" 502 150 2025-11-30 15:53:53,994 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c584498c72996de1fc6b'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:53:54,673 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c586498c72996de1fc6e 2025-11-30 15:53:54,674 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c586498c72996de1fc6e HTTP/1.1" 502 150 2025-11-30 15:53:54,674 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c586498c72996de1fc6e'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:53:55,337 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c588498c72996de1fc71 2025-11-30 15:53:55,338 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c588498c72996de1fc71 HTTP/1.1" 502 150 2025-11-30 15:53:55,338 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c588498c72996de1fc71'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:44502 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:54:00,674 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c580498c72996de1fc65 2025-11-30 15:54:00,676 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c580498c72996de1fc65 HTTP/1.1" 502 150 2025-11-30 15:54:00,676 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c580498c72996de1fc65'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:54:01,310 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c583498c72996de1fc68 2025-11-30 15:54:01,311 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c583498c72996de1fc68 HTTP/1.1" 502 150 2025-11-30 15:54:01,312 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c583498c72996de1fc68'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:54:01,994 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c584498c72996de1fc6b 2025-11-30 15:54:01,995 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c584498c72996de1fc6b HTTP/1.1" 502 150 2025-11-30 15:54:01,995 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c584498c72996de1fc6b'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:54:02,209 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca9fd481a0c6f8cb58b1 2025-11-30 15:54:02,210 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca9fd481a0c6f8cb58b1 HTTP/1.1" 502 150 2025-11-30 15:54:02,210 - lava_callback - ERROR - Error processing callback for node 692aca9fd481a0c6f8cb58b1: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca9fd481a0c6f8cb58b1 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca9fd481a0c6f8cb58b1 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca9fd481a0c6f8cb58b1 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 15:54:02,674 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c586498c72996de1fc6e 2025-11-30 15:54:02,676 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c586498c72996de1fc6e HTTP/1.1" 502 150 2025-11-30 15:54:02,676 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c586498c72996de1fc6e'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:54:03,338 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c588498c72996de1fc71 2025-11-30 15:54:03,339 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c588498c72996de1fc71 HTTP/1.1" 502 150 2025-11-30 15:54:03,339 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c588498c72996de1fc71'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:54:03,494 - lava_callback - INFO - Received callback for node 6929c596498c72996de1fc9a INFO: 172.19.0.1:55992 - "POST /node/6929c596498c72996de1fc9a HTTP/1.0" 202 Accepted 2025-11-30 15:54:03,500 - lava_callback - INFO - Starting async job submit for node 6929c596498c72996de1fc9a 2025-11-30 15:54:03,501 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:54:03,511 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c596498c72996de1fc9a HTTP/1.1" 502 150 2025-11-30 15:54:03,511 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c596498c72996de1fc9a'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:54:03,511 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c596498c72996de1fc9a 2025-11-30 15:54:03,512 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c596498c72996de1fc9a HTTP/1.1" 502 150 2025-11-30 15:54:03,512 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c596498c72996de1fc9a'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:54:04,181 - lava_callback - INFO - Received callback for node 6929c599498c72996de1fc9d INFO: 172.19.0.1:56006 - "POST /node/6929c599498c72996de1fc9d HTTP/1.0" 202 Accepted 2025-11-30 15:54:04,187 - lava_callback - INFO - Starting async job submit for node 6929c599498c72996de1fc9d 2025-11-30 15:54:04,188 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:54:04,201 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c599498c72996de1fc9d HTTP/1.1" 502 150 2025-11-30 15:54:04,201 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c599498c72996de1fc9d'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:54:04,201 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c599498c72996de1fc9d 2025-11-30 15:54:04,202 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c599498c72996de1fc9d HTTP/1.1" 502 150 2025-11-30 15:54:04,202 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c599498c72996de1fc9d'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:54:05,512 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c596498c72996de1fc9a 2025-11-30 15:54:05,513 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c596498c72996de1fc9a HTTP/1.1" 502 150 2025-11-30 15:54:05,513 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c596498c72996de1fc9a'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:54:06,202 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c599498c72996de1fc9d 2025-11-30 15:54:06,203 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c599498c72996de1fc9d HTTP/1.1" 502 150 2025-11-30 15:54:06,204 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c599498c72996de1fc9d'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:56022 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:54:06,614 - lava_callback - INFO - Received callback for node 6929c589498c72996de1fc74 INFO: 172.19.0.1:56034 - "POST /node/6929c589498c72996de1fc74 HTTP/1.0" 202 Accepted 2025-11-30 15:54:06,620 - lava_callback - INFO - Starting async job submit for node 6929c589498c72996de1fc74 2025-11-30 15:54:06,621 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:54:06,630 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c589498c72996de1fc74 HTTP/1.1" 502 150 2025-11-30 15:54:06,630 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c589498c72996de1fc74'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:54:06,631 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c589498c72996de1fc74 2025-11-30 15:54:06,631 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c589498c72996de1fc74 HTTP/1.1" 502 150 2025-11-30 15:54:06,631 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c589498c72996de1fc74'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:54:08,632 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c589498c72996de1fc74 2025-11-30 15:54:08,633 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c589498c72996de1fc74 HTTP/1.1" 502 150 2025-11-30 15:54:08,633 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c589498c72996de1fc74'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:54:09,513 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c596498c72996de1fc9a 2025-11-30 15:54:09,514 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c596498c72996de1fc9a HTTP/1.1" 502 150 2025-11-30 15:54:09,515 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c596498c72996de1fc9a'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:54:10,204 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c599498c72996de1fc9d 2025-11-30 15:54:10,205 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c599498c72996de1fc9d HTTP/1.1" 502 150 2025-11-30 15:54:10,205 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c599498c72996de1fc9d'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:47742 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:54:12,633 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c589498c72996de1fc74 2025-11-30 15:54:12,635 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c589498c72996de1fc74 HTTP/1.1" 502 150 2025-11-30 15:54:12,635 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c589498c72996de1fc74'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:47756 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:47762 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:54:16,676 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c580498c72996de1fc65 2025-11-30 15:54:16,677 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c580498c72996de1fc65 HTTP/1.1" 502 150 2025-11-30 15:54:16,677 - lava_callback - ERROR - Error processing callback for node 6929c580498c72996de1fc65: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c580498c72996de1fc65 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c580498c72996de1fc65 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c580498c72996de1fc65 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 15:54:17,312 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c583498c72996de1fc68 2025-11-30 15:54:17,316 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c583498c72996de1fc68 HTTP/1.1" 502 150 2025-11-30 15:54:17,316 - lava_callback - ERROR - Error processing callback for node 6929c583498c72996de1fc68: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c583498c72996de1fc68 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c583498c72996de1fc68 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c583498c72996de1fc68 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 15:54:17,515 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c596498c72996de1fc9a 2025-11-30 15:54:17,516 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c596498c72996de1fc9a HTTP/1.1" 502 150 2025-11-30 15:54:17,516 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c596498c72996de1fc9a'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:54:17,999 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c584498c72996de1fc6b 2025-11-30 15:54:18,001 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c584498c72996de1fc6b HTTP/1.1" 502 150 2025-11-30 15:54:18,001 - lava_callback - ERROR - Error processing callback for node 6929c584498c72996de1fc6b: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c584498c72996de1fc6b (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c584498c72996de1fc6b (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c584498c72996de1fc6b (Caused by ResponseError('too many 502 error responses')) 2025-11-30 15:54:18,206 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c599498c72996de1fc9d 2025-11-30 15:54:18,207 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c599498c72996de1fc9d HTTP/1.1" 502 150 2025-11-30 15:54:18,207 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c599498c72996de1fc9d'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:54:18,676 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c586498c72996de1fc6e 2025-11-30 15:54:18,678 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c586498c72996de1fc6e HTTP/1.1" 502 150 2025-11-30 15:54:18,678 - lava_callback - ERROR - Error processing callback for node 6929c586498c72996de1fc6e: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c586498c72996de1fc6e (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c586498c72996de1fc6e (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c586498c72996de1fc6e (Caused by ResponseError('too many 502 error responses')) 2025-11-30 15:54:19,339 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c588498c72996de1fc71 2025-11-30 15:54:19,341 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c588498c72996de1fc71 HTTP/1.1" 502 150 2025-11-30 15:54:19,341 - lava_callback - ERROR - Error processing callback for node 6929c588498c72996de1fc71: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c588498c72996de1fc71 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c588498c72996de1fc71 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c588498c72996de1fc71 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 15:54:20,635 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c589498c72996de1fc74 2025-11-30 15:54:20,636 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c589498c72996de1fc74 HTTP/1.1" 502 150 2025-11-30 15:54:20,636 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c589498c72996de1fc74'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:60224 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:54:33,516 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c596498c72996de1fc9a 2025-11-30 15:54:33,518 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c596498c72996de1fc9a HTTP/1.1" 502 150 2025-11-30 15:54:33,518 - lava_callback - ERROR - Error processing callback for node 6929c596498c72996de1fc9a: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c596498c72996de1fc9a (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c596498c72996de1fc9a (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c596498c72996de1fc9a (Caused by ResponseError('too many 502 error responses')) 2025-11-30 15:54:34,208 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c599498c72996de1fc9d 2025-11-30 15:54:34,209 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c599498c72996de1fc9d HTTP/1.1" 502 150 2025-11-30 15:54:34,209 - lava_callback - ERROR - Error processing callback for node 6929c599498c72996de1fc9d: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c599498c72996de1fc9d (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c599498c72996de1fc9d (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c599498c72996de1fc9d (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:57760 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:54:36,636 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c589498c72996de1fc74 2025-11-30 15:54:36,639 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c589498c72996de1fc74 HTTP/1.1" 502 150 2025-11-30 15:54:36,639 - lava_callback - ERROR - Error processing callback for node 6929c589498c72996de1fc74: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c589498c72996de1fc74 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c589498c72996de1fc74 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c589498c72996de1fc74 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:37992 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:57768 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:39984 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:60760 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:60766 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:60770 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:33294 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:37158 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:40280 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:60736 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:39472 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:43634 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:56:14,421 - lava_callback - INFO - Received callback for node 6929c610498c72996de1fcdf INFO: 172.19.0.1:43646 - "POST /node/6929c610498c72996de1fcdf HTTP/1.0" 202 Accepted 2025-11-30 15:56:14,427 - lava_callback - INFO - Starting async job submit for node 6929c610498c72996de1fcdf 2025-11-30 15:56:14,427 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:56:14,462 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c610498c72996de1fcdf HTTP/1.1" 502 150 2025-11-30 15:56:14,462 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c610498c72996de1fcdf'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:56:14,462 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c610498c72996de1fcdf 2025-11-30 15:56:14,463 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c610498c72996de1fcdf HTTP/1.1" 502 150 2025-11-30 15:56:14,463 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c610498c72996de1fcdf'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:43656 - "GET /api/metrics HTTP/1.0" 200 OK 2025-11-30 15:56:16,464 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c610498c72996de1fcdf 2025-11-30 15:56:16,465 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c610498c72996de1fcdf HTTP/1.1" 502 150 2025-11-30 15:56:16,466 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c610498c72996de1fcdf'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:43664 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:56:20,466 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c610498c72996de1fcdf 2025-11-30 15:56:20,467 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c610498c72996de1fcdf HTTP/1.1" 502 150 2025-11-30 15:56:20,467 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c610498c72996de1fcdf'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:46482 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:56:28,467 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c610498c72996de1fcdf 2025-11-30 15:56:28,469 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c610498c72996de1fcdf HTTP/1.1" 502 150 2025-11-30 15:56:28,469 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c610498c72996de1fcdf'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:45838 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:56:44,469 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c610498c72996de1fcdf 2025-11-30 15:56:44,470 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c610498c72996de1fcdf HTTP/1.1" 502 150 2025-11-30 15:56:44,471 - lava_callback - ERROR - Error processing callback for node 6929c610498c72996de1fcdf: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c610498c72996de1fcdf (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c610498c72996de1fcdf (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c610498c72996de1fcdf (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:47538 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:58950 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:53750 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:37092 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:37100 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:37112 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:34944 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:33956 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:43754 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:47260 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:44572 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:34222 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:34238 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:34240 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:54946 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:60928 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:36748 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:58:48,572 - lava_callback - INFO - Received callback for node 692acac4d481a0c6f8cb58bd INFO: 172.19.0.1:36754 - "POST /node/692acac4d481a0c6f8cb58bd HTTP/1.0" 202 Accepted 2025-11-30 15:58:48,579 - lava_callback - INFO - Starting async job submit for node 692acac4d481a0c6f8cb58bd 2025-11-30 15:58:48,611 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:58:48,624 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acac4d481a0c6f8cb58bd HTTP/1.1" 502 150 2025-11-30 15:58:48,624 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acac4d481a0c6f8cb58bd'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:58:48,625 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acac4d481a0c6f8cb58bd 2025-11-30 15:58:48,625 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acac4d481a0c6f8cb58bd HTTP/1.1" 502 150 2025-11-30 15:58:48,625 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acac4d481a0c6f8cb58bd'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:58:50,625 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acac4d481a0c6f8cb58bd 2025-11-30 15:58:50,627 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acac4d481a0c6f8cb58bd HTTP/1.1" 502 150 2025-11-30 15:58:50,627 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acac4d481a0c6f8cb58bd'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:58:54,627 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acac4d481a0c6f8cb58bd 2025-11-30 15:58:54,629 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acac4d481a0c6f8cb58bd HTTP/1.1" 502 150 2025-11-30 15:58:54,629 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acac4d481a0c6f8cb58bd'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:38474 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:59:02,629 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acac4d481a0c6f8cb58bd 2025-11-30 15:59:02,630 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acac4d481a0c6f8cb58bd HTTP/1.1" 502 150 2025-11-30 15:59:02,630 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acac4d481a0c6f8cb58bd'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:39172 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:52580 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:52594 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:52610 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:59:18,630 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acac4d481a0c6f8cb58bd 2025-11-30 15:59:18,631 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acac4d481a0c6f8cb58bd HTTP/1.1" 502 150 2025-11-30 15:59:18,632 - lava_callback - ERROR - Error processing callback for node 692acac4d481a0c6f8cb58bd: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acac4d481a0c6f8cb58bd (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acac4d481a0c6f8cb58bd (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acac4d481a0c6f8cb58bd (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:36862 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:59:32,348 - lava_callback - INFO - Received callback for node 692acaa0d481a0c6f8cb58b4 INFO: 172.19.0.1:46086 - "POST /node/692acaa0d481a0c6f8cb58b4 HTTP/1.0" 202 Accepted 2025-11-30 15:59:32,356 - lava_callback - INFO - Starting async job submit for node 692acaa0d481a0c6f8cb58b4 2025-11-30 15:59:32,397 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 15:59:32,417 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acaa0d481a0c6f8cb58b4 HTTP/1.1" 502 150 2025-11-30 15:59:32,417 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acaa0d481a0c6f8cb58b4'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:59:32,417 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acaa0d481a0c6f8cb58b4 2025-11-30 15:59:32,418 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acaa0d481a0c6f8cb58b4 HTTP/1.1" 502 150 2025-11-30 15:59:32,418 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acaa0d481a0c6f8cb58b4'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:59:34,418 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acaa0d481a0c6f8cb58b4 2025-11-30 15:59:34,420 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acaa0d481a0c6f8cb58b4 HTTP/1.1" 502 150 2025-11-30 15:59:34,420 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acaa0d481a0c6f8cb58b4'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:46092 - "GET / HTTP/1.0" 200 OK 2025-11-30 15:59:38,420 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acaa0d481a0c6f8cb58b4 2025-11-30 15:59:38,421 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acaa0d481a0c6f8cb58b4 HTTP/1.1" 502 150 2025-11-30 15:59:38,421 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acaa0d481a0c6f8cb58b4'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 15:59:46,421 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acaa0d481a0c6f8cb58b4 2025-11-30 15:59:46,422 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acaa0d481a0c6f8cb58b4 HTTP/1.1" 502 150 2025-11-30 15:59:46,422 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acaa0d481a0c6f8cb58b4'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:59994 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:33728 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:00:02,423 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acaa0d481a0c6f8cb58b4 2025-11-30 16:00:02,424 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acaa0d481a0c6f8cb58b4 HTTP/1.1" 502 150 2025-11-30 16:00:02,424 - lava_callback - ERROR - Error processing callback for node 692acaa0d481a0c6f8cb58b4: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acaa0d481a0c6f8cb58b4 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acaa0d481a0c6f8cb58b4 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acaa0d481a0c6f8cb58b4 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:59656 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:00:06,944 - lava_callback - INFO - Received callback for node 6929c704498c72996de1fd09 INFO: 172.19.0.1:59662 - "POST /node/6929c704498c72996de1fd09 HTTP/1.0" 202 Accepted 2025-11-30 16:00:06,950 - lava_callback - INFO - Starting async job submit for node 6929c704498c72996de1fd09 2025-11-30 16:00:06,951 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:00:06,961 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c704498c72996de1fd09 HTTP/1.1" 502 150 2025-11-30 16:00:06,961 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c704498c72996de1fd09'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:06,961 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c704498c72996de1fd09 2025-11-30 16:00:06,962 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c704498c72996de1fd09 HTTP/1.1" 502 150 2025-11-30 16:00:06,962 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c704498c72996de1fd09'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:07,590 - lava_callback - INFO - Received callback for node 6929c701498c72996de1fd06 INFO: 172.19.0.1:59668 - "POST /node/6929c701498c72996de1fd06 HTTP/1.0" 202 Accepted 2025-11-30 16:00:07,596 - lava_callback - INFO - Starting async job submit for node 6929c701498c72996de1fd06 2025-11-30 16:00:07,597 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:00:07,607 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c701498c72996de1fd06 HTTP/1.1" 502 150 2025-11-30 16:00:07,607 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c701498c72996de1fd06'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:07,607 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c701498c72996de1fd06 2025-11-30 16:00:07,608 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c701498c72996de1fd06 HTTP/1.1" 502 150 2025-11-30 16:00:07,608 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c701498c72996de1fd06'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:08,962 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c704498c72996de1fd09 2025-11-30 16:00:08,963 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c704498c72996de1fd09 HTTP/1.1" 502 150 2025-11-30 16:00:08,963 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c704498c72996de1fd09'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:09,608 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c701498c72996de1fd06 2025-11-30 16:00:09,609 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c701498c72996de1fd06 HTTP/1.1" 502 150 2025-11-30 16:00:09,609 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c701498c72996de1fd06'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:60638 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:00:12,964 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c704498c72996de1fd09 2025-11-30 16:00:12,965 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c704498c72996de1fd09 HTTP/1.1" 502 150 2025-11-30 16:00:12,965 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c704498c72996de1fd09'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:13,610 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c701498c72996de1fd06 2025-11-30 16:00:13,611 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c701498c72996de1fd06 HTTP/1.1" 502 150 2025-11-30 16:00:13,611 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c701498c72996de1fd06'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:60642 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:60646 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:00:20,965 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c704498c72996de1fd09 2025-11-30 16:00:20,966 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c704498c72996de1fd09 HTTP/1.1" 502 150 2025-11-30 16:00:20,966 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c704498c72996de1fd09'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:21,611 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c701498c72996de1fd06 2025-11-30 16:00:21,612 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c701498c72996de1fd06 HTTP/1.1" 502 150 2025-11-30 16:00:21,612 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c701498c72996de1fd06'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:35126 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:00:26,925 - lava_callback - INFO - Received callback for node 6929c70d498c72996de1fd12 INFO: 172.19.0.1:35134 - "POST /node/6929c70d498c72996de1fd12 HTTP/1.0" 202 Accepted 2025-11-30 16:00:26,931 - lava_callback - INFO - Starting async job submit for node 6929c70d498c72996de1fd12 2025-11-30 16:00:26,932 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:00:26,950 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c70d498c72996de1fd12 HTTP/1.1" 502 150 2025-11-30 16:00:26,950 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c70d498c72996de1fd12'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:26,950 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c70d498c72996de1fd12 2025-11-30 16:00:26,951 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c70d498c72996de1fd12 HTTP/1.1" 502 150 2025-11-30 16:00:26,951 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c70d498c72996de1fd12'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:27,579 - lava_callback - INFO - Received callback for node 6929c707498c72996de1fd0c INFO: 172.19.0.1:35140 - "POST /node/6929c707498c72996de1fd0c HTTP/1.0" 202 Accepted 2025-11-30 16:00:27,585 - lava_callback - INFO - Starting async job submit for node 6929c707498c72996de1fd0c 2025-11-30 16:00:27,586 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:00:27,599 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c707498c72996de1fd0c HTTP/1.1" 502 150 2025-11-30 16:00:27,600 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c707498c72996de1fd0c'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:27,600 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c707498c72996de1fd0c 2025-11-30 16:00:27,600 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c707498c72996de1fd0c HTTP/1.1" 502 150 2025-11-30 16:00:27,600 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c707498c72996de1fd0c'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:28,241 - lava_callback - INFO - Received callback for node 6929c70a498c72996de1fd0f INFO: 172.19.0.1:35144 - "POST /node/6929c70a498c72996de1fd0f HTTP/1.0" 202 Accepted 2025-11-30 16:00:28,247 - lava_callback - INFO - Starting async job submit for node 6929c70a498c72996de1fd0f 2025-11-30 16:00:28,248 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:00:28,279 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c70a498c72996de1fd0f HTTP/1.1" 502 150 2025-11-30 16:00:28,280 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c70a498c72996de1fd0f'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:28,280 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c70a498c72996de1fd0f 2025-11-30 16:00:28,280 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c70a498c72996de1fd0f HTTP/1.1" 502 150 2025-11-30 16:00:28,280 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c70a498c72996de1fd0f'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:28,907 - lava_callback - INFO - Received callback for node 6929c70f498c72996de1fd15 INFO: 172.19.0.1:35148 - "POST /node/6929c70f498c72996de1fd15 HTTP/1.0" 202 Accepted 2025-11-30 16:00:28,912 - lava_callback - INFO - Starting async job submit for node 6929c70f498c72996de1fd15 2025-11-30 16:00:28,913 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:00:28,923 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c70f498c72996de1fd15 HTTP/1.1" 502 150 2025-11-30 16:00:28,924 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c70f498c72996de1fd15'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:28,924 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c70f498c72996de1fd15 2025-11-30 16:00:28,925 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c70f498c72996de1fd15 HTTP/1.1" 502 150 2025-11-30 16:00:28,925 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c70f498c72996de1fd15'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:28,951 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c70d498c72996de1fd12 2025-11-30 16:00:28,952 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c70d498c72996de1fd12 HTTP/1.1" 502 150 2025-11-30 16:00:28,952 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c70d498c72996de1fd12'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:29,574 - lava_callback - INFO - Received callback for node 6929c712498c72996de1fd18 INFO: 172.19.0.1:35160 - "POST /node/6929c712498c72996de1fd18 HTTP/1.0" 202 Accepted 2025-11-30 16:00:29,583 - lava_callback - INFO - Starting async job submit for node 6929c712498c72996de1fd18 2025-11-30 16:00:29,584 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:00:29,601 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c707498c72996de1fd0c 2025-11-30 16:00:29,602 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c707498c72996de1fd0c HTTP/1.1" 502 150 2025-11-30 16:00:29,602 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c707498c72996de1fd0c'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:29,620 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c712498c72996de1fd18 HTTP/1.1" 502 150 2025-11-30 16:00:29,620 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c712498c72996de1fd18'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:29,620 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c712498c72996de1fd18 2025-11-30 16:00:29,621 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c712498c72996de1fd18 HTTP/1.1" 502 150 2025-11-30 16:00:29,621 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c712498c72996de1fd18'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:30,257 - lava_callback - INFO - Received callback for node 6929c715498c72996de1fd1b INFO: 172.19.0.1:56606 - "POST /node/6929c715498c72996de1fd1b HTTP/1.0" 202 Accepted 2025-11-30 16:00:30,262 - lava_callback - INFO - Starting async job submit for node 6929c715498c72996de1fd1b 2025-11-30 16:00:30,263 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:00:30,273 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c715498c72996de1fd1b HTTP/1.1" 502 150 2025-11-30 16:00:30,274 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c715498c72996de1fd1b'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:30,274 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c715498c72996de1fd1b 2025-11-30 16:00:30,274 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c715498c72996de1fd1b HTTP/1.1" 502 150 2025-11-30 16:00:30,275 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c715498c72996de1fd1b'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:30,281 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c70a498c72996de1fd0f 2025-11-30 16:00:30,281 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c70a498c72996de1fd0f HTTP/1.1" 502 150 2025-11-30 16:00:30,282 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c70a498c72996de1fd0f'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:30,915 - lava_callback - INFO - Received callback for node 6929c718498c72996de1fd1e INFO: 172.19.0.1:56622 - "POST /node/6929c718498c72996de1fd1e HTTP/1.0" 202 Accepted 2025-11-30 16:00:30,920 - lava_callback - INFO - Starting async job submit for node 6929c718498c72996de1fd1e 2025-11-30 16:00:30,921 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:00:30,925 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c70f498c72996de1fd15 2025-11-30 16:00:30,926 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c70f498c72996de1fd15 HTTP/1.1" 502 150 2025-11-30 16:00:30,926 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c70f498c72996de1fd15'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:30,935 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c718498c72996de1fd1e HTTP/1.1" 502 150 2025-11-30 16:00:30,935 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c718498c72996de1fd1e'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:30,935 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c718498c72996de1fd1e 2025-11-30 16:00:30,936 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c718498c72996de1fd1e HTTP/1.1" 502 150 2025-11-30 16:00:30,936 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c718498c72996de1fd1e'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:31,622 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c712498c72996de1fd18 2025-11-30 16:00:31,623 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c712498c72996de1fd18 HTTP/1.1" 502 150 2025-11-30 16:00:31,623 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c712498c72996de1fd18'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:32,275 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c715498c72996de1fd1b 2025-11-30 16:00:32,276 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c715498c72996de1fd1b HTTP/1.1" 502 150 2025-11-30 16:00:32,276 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c715498c72996de1fd1b'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:32,936 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c718498c72996de1fd1e 2025-11-30 16:00:32,937 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c718498c72996de1fd1e HTTP/1.1" 502 150 2025-11-30 16:00:32,937 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c718498c72996de1fd1e'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:32,953 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c70d498c72996de1fd12 2025-11-30 16:00:32,953 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c70d498c72996de1fd12 HTTP/1.1" 502 150 2025-11-30 16:00:32,954 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c70d498c72996de1fd12'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:33,602 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c707498c72996de1fd0c 2025-11-30 16:00:33,603 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c707498c72996de1fd0c HTTP/1.1" 502 150 2025-11-30 16:00:33,603 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c707498c72996de1fd0c'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:34,282 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c70a498c72996de1fd0f 2025-11-30 16:00:34,283 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c70a498c72996de1fd0f HTTP/1.1" 502 150 2025-11-30 16:00:34,284 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c70a498c72996de1fd0f'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:34,926 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c70f498c72996de1fd15 2025-11-30 16:00:34,927 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c70f498c72996de1fd15 HTTP/1.1" 502 150 2025-11-30 16:00:34,927 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c70f498c72996de1fd15'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:35,623 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c712498c72996de1fd18 2025-11-30 16:00:35,624 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c712498c72996de1fd18 HTTP/1.1" 502 150 2025-11-30 16:00:35,624 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c712498c72996de1fd18'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:36,276 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c715498c72996de1fd1b 2025-11-30 16:00:36,277 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c715498c72996de1fd1b HTTP/1.1" 502 150 2025-11-30 16:00:36,277 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c715498c72996de1fd1b'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:56636 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:00:36,938 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c718498c72996de1fd1e 2025-11-30 16:00:36,939 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c718498c72996de1fd1e HTTP/1.1" 502 150 2025-11-30 16:00:36,939 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c718498c72996de1fd1e'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:36,967 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c704498c72996de1fd09 2025-11-30 16:00:36,968 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c704498c72996de1fd09 HTTP/1.1" 502 150 2025-11-30 16:00:36,968 - lava_callback - ERROR - Error processing callback for node 6929c704498c72996de1fd09: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c704498c72996de1fd09 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c704498c72996de1fd09 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c704498c72996de1fd09 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:00:37,072 - lava_callback - INFO - Received callback for node 6929c71b498c72996de1fd21 INFO: 172.19.0.1:56646 - "POST /node/6929c71b498c72996de1fd21 HTTP/1.0" 202 Accepted 2025-11-30 16:00:37,079 - lava_callback - INFO - Starting async job submit for node 6929c71b498c72996de1fd21 2025-11-30 16:00:37,080 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:00:37,120 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c71b498c72996de1fd21 HTTP/1.1" 502 150 2025-11-30 16:00:37,120 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c71b498c72996de1fd21'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:37,120 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c71b498c72996de1fd21 2025-11-30 16:00:37,121 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c71b498c72996de1fd21 HTTP/1.1" 502 150 2025-11-30 16:00:37,121 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c71b498c72996de1fd21'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:37,613 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c701498c72996de1fd06 2025-11-30 16:00:37,614 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c701498c72996de1fd06 HTTP/1.1" 502 150 2025-11-30 16:00:37,614 - lava_callback - ERROR - Error processing callback for node 6929c701498c72996de1fd06: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c701498c72996de1fd06 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c701498c72996de1fd06 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c701498c72996de1fd06 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:00:37,687 - lava_callback - INFO - Received callback for node 6929c71e498c72996de1fd24 INFO: 172.19.0.1:56656 - "POST /node/6929c71e498c72996de1fd24 HTTP/1.0" 202 Accepted 2025-11-30 16:00:37,693 - lava_callback - INFO - Starting async job submit for node 6929c71e498c72996de1fd24 2025-11-30 16:00:37,694 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:00:37,705 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c71e498c72996de1fd24 HTTP/1.1" 502 150 2025-11-30 16:00:37,705 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c71e498c72996de1fd24'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:37,705 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c71e498c72996de1fd24 2025-11-30 16:00:37,706 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c71e498c72996de1fd24 HTTP/1.1" 502 150 2025-11-30 16:00:37,706 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c71e498c72996de1fd24'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:39,121 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c71b498c72996de1fd21 2025-11-30 16:00:39,123 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c71b498c72996de1fd21 HTTP/1.1" 502 150 2025-11-30 16:00:39,123 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c71b498c72996de1fd21'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:39,707 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c71e498c72996de1fd24 2025-11-30 16:00:39,708 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c71e498c72996de1fd24 HTTP/1.1" 502 150 2025-11-30 16:00:39,708 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c71e498c72996de1fd24'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:40,954 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c70d498c72996de1fd12 2025-11-30 16:00:40,955 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c70d498c72996de1fd12 HTTP/1.1" 502 150 2025-11-30 16:00:40,955 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c70d498c72996de1fd12'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:41,603 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c707498c72996de1fd0c 2025-11-30 16:00:41,605 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c707498c72996de1fd0c HTTP/1.1" 502 150 2025-11-30 16:00:41,605 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c707498c72996de1fd0c'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:42,284 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c70a498c72996de1fd0f 2025-11-30 16:00:42,285 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c70a498c72996de1fd0f HTTP/1.1" 502 150 2025-11-30 16:00:42,285 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c70a498c72996de1fd0f'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:42,927 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c70f498c72996de1fd15 2025-11-30 16:00:42,929 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c70f498c72996de1fd15 HTTP/1.1" 502 150 2025-11-30 16:00:42,929 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c70f498c72996de1fd15'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:43,123 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c71b498c72996de1fd21 2025-11-30 16:00:43,124 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c71b498c72996de1fd21 HTTP/1.1" 502 150 2025-11-30 16:00:43,124 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c71b498c72996de1fd21'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:43,625 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c712498c72996de1fd18 2025-11-30 16:00:43,626 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c712498c72996de1fd18 HTTP/1.1" 502 150 2025-11-30 16:00:43,626 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c712498c72996de1fd18'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:43,708 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c71e498c72996de1fd24 2025-11-30 16:00:43,709 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c71e498c72996de1fd24 HTTP/1.1" 502 150 2025-11-30 16:00:43,709 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c71e498c72996de1fd24'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:44,278 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c715498c72996de1fd1b 2025-11-30 16:00:44,279 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c715498c72996de1fd1b HTTP/1.1" 502 150 2025-11-30 16:00:44,279 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c715498c72996de1fd1b'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:44,939 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c718498c72996de1fd1e 2025-11-30 16:00:44,940 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c718498c72996de1fd1e HTTP/1.1" 502 150 2025-11-30 16:00:44,940 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c718498c72996de1fd1e'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:53340 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:00:51,124 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c71b498c72996de1fd21 2025-11-30 16:00:51,125 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c71b498c72996de1fd21 HTTP/1.1" 502 150 2025-11-30 16:00:51,126 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c71b498c72996de1fd21'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:51,709 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c71e498c72996de1fd24 2025-11-30 16:00:51,711 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c71e498c72996de1fd24 HTTP/1.1" 502 150 2025-11-30 16:00:51,711 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c71e498c72996de1fd24'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:52,629 - lava_callback - INFO - Received callback for node 692aca98d481a0c6f8cb58a5 INFO: 172.19.0.1:45476 - "POST /node/692aca98d481a0c6f8cb58a5 HTTP/1.0" 202 Accepted 2025-11-30 16:00:52,636 - lava_callback - INFO - Starting async job submit for node 692aca98d481a0c6f8cb58a5 2025-11-30 16:00:52,766 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:00:52,780 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca98d481a0c6f8cb58a5 HTTP/1.1" 502 150 2025-11-30 16:00:52,780 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca98d481a0c6f8cb58a5'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:52,780 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca98d481a0c6f8cb58a5 2025-11-30 16:00:52,781 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca98d481a0c6f8cb58a5 HTTP/1.1" 502 150 2025-11-30 16:00:52,781 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca98d481a0c6f8cb58a5'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:54,781 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca98d481a0c6f8cb58a5 2025-11-30 16:00:54,782 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca98d481a0c6f8cb58a5 HTTP/1.1" 502 150 2025-11-30 16:00:54,782 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca98d481a0c6f8cb58a5'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:45488 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:00:56,955 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c70d498c72996de1fd12 2025-11-30 16:00:56,957 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c70d498c72996de1fd12 HTTP/1.1" 502 150 2025-11-30 16:00:56,957 - lava_callback - ERROR - Error processing callback for node 6929c70d498c72996de1fd12: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c70d498c72996de1fd12 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c70d498c72996de1fd12 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c70d498c72996de1fd12 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:00:57,605 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c707498c72996de1fd0c 2025-11-30 16:00:57,607 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c707498c72996de1fd0c HTTP/1.1" 502 150 2025-11-30 16:00:57,607 - lava_callback - ERROR - Error processing callback for node 6929c707498c72996de1fd0c: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c707498c72996de1fd0c (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c707498c72996de1fd0c (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c707498c72996de1fd0c (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:00:58,285 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c70a498c72996de1fd0f 2025-11-30 16:00:58,287 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c70a498c72996de1fd0f HTTP/1.1" 502 150 2025-11-30 16:00:58,287 - lava_callback - ERROR - Error processing callback for node 6929c70a498c72996de1fd0f: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c70a498c72996de1fd0f (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c70a498c72996de1fd0f (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c70a498c72996de1fd0f (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:00:58,782 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca98d481a0c6f8cb58a5 2025-11-30 16:00:58,783 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca98d481a0c6f8cb58a5 HTTP/1.1" 502 150 2025-11-30 16:00:58,784 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca98d481a0c6f8cb58a5'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:00:58,929 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c70f498c72996de1fd15 2025-11-30 16:00:58,930 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c70f498c72996de1fd15 HTTP/1.1" 502 150 2025-11-30 16:00:58,930 - lava_callback - ERROR - Error processing callback for node 6929c70f498c72996de1fd15: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c70f498c72996de1fd15 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c70f498c72996de1fd15 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c70f498c72996de1fd15 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:00:59,626 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c712498c72996de1fd18 2025-11-30 16:00:59,628 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c712498c72996de1fd18 HTTP/1.1" 502 150 2025-11-30 16:00:59,628 - lava_callback - ERROR - Error processing callback for node 6929c712498c72996de1fd18: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c712498c72996de1fd18 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c712498c72996de1fd18 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c712498c72996de1fd18 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:01:00,279 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c715498c72996de1fd1b 2025-11-30 16:01:00,281 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c715498c72996de1fd1b HTTP/1.1" 502 150 2025-11-30 16:01:00,281 - lava_callback - ERROR - Error processing callback for node 6929c715498c72996de1fd1b: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c715498c72996de1fd1b (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c715498c72996de1fd1b (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c715498c72996de1fd1b (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:01:00,945 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c718498c72996de1fd1e 2025-11-30 16:01:00,946 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c718498c72996de1fd1e HTTP/1.1" 502 150 2025-11-30 16:01:00,946 - lava_callback - ERROR - Error processing callback for node 6929c718498c72996de1fd1e: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c718498c72996de1fd1e (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c718498c72996de1fd1e (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c718498c72996de1fd1e (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:38924 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:01:06,784 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca98d481a0c6f8cb58a5 2025-11-30 16:01:06,785 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca98d481a0c6f8cb58a5 HTTP/1.1" 502 150 2025-11-30 16:01:06,785 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692aca98d481a0c6f8cb58a5'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:01:07,126 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c71b498c72996de1fd21 2025-11-30 16:01:07,127 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c71b498c72996de1fd21 HTTP/1.1" 502 150 2025-11-30 16:01:07,128 - lava_callback - ERROR - Error processing callback for node 6929c71b498c72996de1fd21: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c71b498c72996de1fd21 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c71b498c72996de1fd21 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c71b498c72996de1fd21 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:01:07,711 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c71e498c72996de1fd24 2025-11-30 16:01:07,712 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c71e498c72996de1fd24 HTTP/1.1" 502 150 2025-11-30 16:01:07,712 - lava_callback - ERROR - Error processing callback for node 6929c71e498c72996de1fd24: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c71e498c72996de1fd24 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c71e498c72996de1fd24 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c71e498c72996de1fd24 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:55712 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:01:12,696 - lava_callback - INFO - Received callback for node 692acac9d481a0c6f8cb58c6 INFO: 172.19.0.1:55718 - "POST /node/692acac9d481a0c6f8cb58c6 HTTP/1.0" 202 Accepted 2025-11-30 16:01:12,708 - lava_callback - INFO - Starting async job submit for node 692acac9d481a0c6f8cb58c6 2025-11-30 16:01:12,906 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:01:12,948 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acac9d481a0c6f8cb58c6 HTTP/1.1" 502 150 2025-11-30 16:01:12,948 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acac9d481a0c6f8cb58c6'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:01:12,948 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acac9d481a0c6f8cb58c6 2025-11-30 16:01:12,949 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acac9d481a0c6f8cb58c6 HTTP/1.1" 502 150 2025-11-30 16:01:12,949 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acac9d481a0c6f8cb58c6'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:01:14,950 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acac9d481a0c6f8cb58c6 2025-11-30 16:01:14,957 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acac9d481a0c6f8cb58c6 HTTP/1.1" 502 150 2025-11-30 16:01:14,957 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acac9d481a0c6f8cb58c6'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:55734 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:55746 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:01:18,958 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acac9d481a0c6f8cb58c6 2025-11-30 16:01:18,959 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acac9d481a0c6f8cb58c6 HTTP/1.1" 502 150 2025-11-30 16:01:18,959 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acac9d481a0c6f8cb58c6'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:01:22,785 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692aca98d481a0c6f8cb58a5 2025-11-30 16:01:22,786 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692aca98d481a0c6f8cb58a5 HTTP/1.1" 502 150 2025-11-30 16:01:22,787 - lava_callback - ERROR - Error processing callback for node 692aca98d481a0c6f8cb58a5: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca98d481a0c6f8cb58a5 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca98d481a0c6f8cb58a5 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692aca98d481a0c6f8cb58a5 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:58342 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:01:26,959 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acac9d481a0c6f8cb58c6 2025-11-30 16:01:26,960 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acac9d481a0c6f8cb58c6 HTTP/1.1" 502 150 2025-11-30 16:01:26,961 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acac9d481a0c6f8cb58c6'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:39164 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:01:42,961 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acac9d481a0c6f8cb58c6 2025-11-30 16:01:42,962 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acac9d481a0c6f8cb58c6 HTTP/1.1" 502 150 2025-11-30 16:01:42,962 - lava_callback - ERROR - Error processing callback for node 692acac9d481a0c6f8cb58c6: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acac9d481a0c6f8cb58c6 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acac9d481a0c6f8cb58c6 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acac9d481a0c6f8cb58c6 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:36030 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:35924 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:57010 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:33962 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:33978 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:33990 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:43264 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:34520 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:38174 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:02:49,051 - lava_callback - INFO - Received callback for node 692acacad481a0c6f8cb58c9 INFO: 172.19.0.1:38176 - "POST /node/692acacad481a0c6f8cb58c9 HTTP/1.0" 202 Accepted 2025-11-30 16:02:49,058 - lava_callback - INFO - Starting async job submit for node 692acacad481a0c6f8cb58c9 2025-11-30 16:02:49,083 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:02:49,093 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acacad481a0c6f8cb58c9 HTTP/1.1" 502 150 2025-11-30 16:02:49,093 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acacad481a0c6f8cb58c9'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:02:49,093 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acacad481a0c6f8cb58c9 2025-11-30 16:02:49,093 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acacad481a0c6f8cb58c9 HTTP/1.1" 502 150 2025-11-30 16:02:49,093 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acacad481a0c6f8cb58c9'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:02:51,094 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acacad481a0c6f8cb58c9 2025-11-30 16:02:51,095 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acacad481a0c6f8cb58c9 HTTP/1.1" 502 150 2025-11-30 16:02:51,095 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acacad481a0c6f8cb58c9'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:02:55,096 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acacad481a0c6f8cb58c9 2025-11-30 16:02:55,097 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acacad481a0c6f8cb58c9 HTTP/1.1" 502 150 2025-11-30 16:02:55,097 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acacad481a0c6f8cb58c9'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:57124 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:03:03,097 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acacad481a0c6f8cb58c9 2025-11-30 16:03:03,098 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acacad481a0c6f8cb58c9 HTTP/1.1" 502 150 2025-11-30 16:03:03,098 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acacad481a0c6f8cb58c9'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:50516 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:03:12,358 - lava_callback - INFO - Received callback for node 692acf09d481a0c6f8cb591f INFO: 172.19.0.1:32926 - "POST /node/692acf09d481a0c6f8cb591f HTTP/1.0" 202 Accepted 2025-11-30 16:03:12,364 - lava_callback - INFO - Starting async job submit for node 692acf09d481a0c6f8cb591f 2025-11-30 16:03:12,386 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:03:12,400 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf09d481a0c6f8cb591f HTTP/1.1" 502 150 2025-11-30 16:03:12,400 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf09d481a0c6f8cb591f'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:03:12,400 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf09d481a0c6f8cb591f 2025-11-30 16:03:12,401 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf09d481a0c6f8cb591f HTTP/1.1" 502 150 2025-11-30 16:03:12,401 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf09d481a0c6f8cb591f'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:32928 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:03:14,401 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf09d481a0c6f8cb591f 2025-11-30 16:03:14,404 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf09d481a0c6f8cb591f HTTP/1.1" 502 150 2025-11-30 16:03:14,404 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf09d481a0c6f8cb591f'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:32944 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:32946 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:03:18,404 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf09d481a0c6f8cb591f 2025-11-30 16:03:18,406 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf09d481a0c6f8cb591f HTTP/1.1" 502 150 2025-11-30 16:03:18,406 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf09d481a0c6f8cb591f'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:03:19,098 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acacad481a0c6f8cb58c9 2025-11-30 16:03:19,100 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acacad481a0c6f8cb58c9 HTTP/1.1" 502 150 2025-11-30 16:03:19,100 - lava_callback - ERROR - Error processing callback for node 692acacad481a0c6f8cb58c9: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acacad481a0c6f8cb58c9 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acacad481a0c6f8cb58c9 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acacad481a0c6f8cb58c9 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:03:26,406 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf09d481a0c6f8cb591f 2025-11-30 16:03:26,407 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf09d481a0c6f8cb591f HTTP/1.1" 502 150 2025-11-30 16:03:26,407 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf09d481a0c6f8cb591f'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:38048 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:03:32,983 - lava_callback - INFO - Received callback for node 692acac5d481a0c6f8cb58c0 INFO: 172.19.0.1:35222 - "POST /node/692acac5d481a0c6f8cb58c0 HTTP/1.0" 202 Accepted 2025-11-30 16:03:32,991 - lava_callback - INFO - Starting async job submit for node 692acac5d481a0c6f8cb58c0 2025-11-30 16:03:33,054 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:03:33,065 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acac5d481a0c6f8cb58c0 HTTP/1.1" 502 150 2025-11-30 16:03:33,065 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acac5d481a0c6f8cb58c0'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:03:33,066 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acac5d481a0c6f8cb58c0 2025-11-30 16:03:33,066 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acac5d481a0c6f8cb58c0 HTTP/1.1" 502 150 2025-11-30 16:03:33,066 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acac5d481a0c6f8cb58c0'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:03:34,622 - lava_callback - INFO - Received callback for node 692acaccd481a0c6f8cb58cc INFO: 172.19.0.1:35230 - "POST /node/692acaccd481a0c6f8cb58cc HTTP/1.0" 202 Accepted 2025-11-30 16:03:34,629 - lava_callback - INFO - Starting async job submit for node 692acaccd481a0c6f8cb58cc 2025-11-30 16:03:34,662 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:03:34,672 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acaccd481a0c6f8cb58cc HTTP/1.1" 502 150 2025-11-30 16:03:34,672 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acaccd481a0c6f8cb58cc'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:03:34,673 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acaccd481a0c6f8cb58cc 2025-11-30 16:03:34,673 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acaccd481a0c6f8cb58cc HTTP/1.1" 502 150 2025-11-30 16:03:34,673 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acaccd481a0c6f8cb58cc'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:03:35,067 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acac5d481a0c6f8cb58c0 2025-11-30 16:03:35,068 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acac5d481a0c6f8cb58c0 HTTP/1.1" 502 150 2025-11-30 16:03:35,068 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acac5d481a0c6f8cb58c0'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:35242 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:03:36,674 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acaccd481a0c6f8cb58cc 2025-11-30 16:03:36,675 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acaccd481a0c6f8cb58cc HTTP/1.1" 502 150 2025-11-30 16:03:36,675 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acaccd481a0c6f8cb58cc'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:03:39,069 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acac5d481a0c6f8cb58c0 2025-11-30 16:03:39,070 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acac5d481a0c6f8cb58c0 HTTP/1.1" 502 150 2025-11-30 16:03:39,070 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acac5d481a0c6f8cb58c0'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:03:40,675 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acaccd481a0c6f8cb58cc 2025-11-30 16:03:40,676 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acaccd481a0c6f8cb58cc HTTP/1.1" 502 150 2025-11-30 16:03:40,676 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acaccd481a0c6f8cb58cc'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:03:42,408 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf09d481a0c6f8cb591f 2025-11-30 16:03:42,409 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf09d481a0c6f8cb591f HTTP/1.1" 502 150 2025-11-30 16:03:42,409 - lava_callback - ERROR - Error processing callback for node 692acf09d481a0c6f8cb591f: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf09d481a0c6f8cb591f (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf09d481a0c6f8cb591f (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf09d481a0c6f8cb591f (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:55120 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:03:47,070 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acac5d481a0c6f8cb58c0 2025-11-30 16:03:47,073 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acac5d481a0c6f8cb58c0 HTTP/1.1" 502 150 2025-11-30 16:03:47,073 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acac5d481a0c6f8cb58c0'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:03:48,676 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acaccd481a0c6f8cb58cc 2025-11-30 16:03:48,677 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acaccd481a0c6f8cb58cc HTTP/1.1" 502 150 2025-11-30 16:03:48,677 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acaccd481a0c6f8cb58cc'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:45654 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:04:03,073 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acac5d481a0c6f8cb58c0 2025-11-30 16:04:03,075 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acac5d481a0c6f8cb58c0 HTTP/1.1" 502 150 2025-11-30 16:04:03,075 - lava_callback - ERROR - Error processing callback for node 692acac5d481a0c6f8cb58c0: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acac5d481a0c6f8cb58c0 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acac5d481a0c6f8cb58c0 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acac5d481a0c6f8cb58c0 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:04:04,678 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acaccd481a0c6f8cb58cc 2025-11-30 16:04:04,679 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acaccd481a0c6f8cb58cc HTTP/1.1" 502 150 2025-11-30 16:04:04,679 - lava_callback - ERROR - Error processing callback for node 692acaccd481a0c6f8cb58cc: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acaccd481a0c6f8cb58cc (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acaccd481a0c6f8cb58cc (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acaccd481a0c6f8cb58cc (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:58652 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:33888 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:33892 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:33900 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:48532 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:48500 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:51868 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:41272 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:60094 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:05:12,572 - lava_callback - INFO - Received callback for node 692acf3ed481a0c6f8cb5947 INFO: 172.19.0.1:34588 - "POST /node/692acf3ed481a0c6f8cb5947 HTTP/1.0" 202 Accepted 2025-11-30 16:05:12,578 - lava_callback - INFO - Starting async job submit for node 692acf3ed481a0c6f8cb5947 2025-11-30 16:05:12,603 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 INFO: 172.19.0.1:34596 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:05:12,632 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf3ed481a0c6f8cb5947 HTTP/1.1" 502 150 2025-11-30 16:05:12,632 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf3ed481a0c6f8cb5947'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:05:12,632 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf3ed481a0c6f8cb5947 2025-11-30 16:05:12,633 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf3ed481a0c6f8cb5947 HTTP/1.1" 502 150 2025-11-30 16:05:12,633 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf3ed481a0c6f8cb5947'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:05:14,633 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf3ed481a0c6f8cb5947 2025-11-30 16:05:14,634 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf3ed481a0c6f8cb5947 HTTP/1.1" 502 150 2025-11-30 16:05:14,634 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf3ed481a0c6f8cb5947'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:34608 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:34612 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:05:18,634 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf3ed481a0c6f8cb5947 2025-11-30 16:05:18,635 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf3ed481a0c6f8cb5947 HTTP/1.1" 502 150 2025-11-30 16:05:18,636 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf3ed481a0c6f8cb5947'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:43798 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:05:26,636 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf3ed481a0c6f8cb5947 2025-11-30 16:05:26,637 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf3ed481a0c6f8cb5947 HTTP/1.1" 502 150 2025-11-30 16:05:26,637 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf3ed481a0c6f8cb5947'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:39194 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:05:42,637 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf3ed481a0c6f8cb5947 2025-11-30 16:05:42,638 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf3ed481a0c6f8cb5947 HTTP/1.1" 502 150 2025-11-30 16:05:42,639 - lava_callback - ERROR - Error processing callback for node 692acf3ed481a0c6f8cb5947: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf3ed481a0c6f8cb5947 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf3ed481a0c6f8cb5947 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf3ed481a0c6f8cb5947 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:43600 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:48000 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:46390 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:56826 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:56836 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:56844 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:06:17,276 - lava_callback - INFO - Received callback for node 692a058a7d8b615fd0b02930 INFO: 172.19.0.1:56850 - "POST /node/692a058a7d8b615fd0b02930 HTTP/1.0" 202 Accepted 2025-11-30 16:06:17,283 - lava_callback - INFO - Starting async job submit for node 692a058a7d8b615fd0b02930 2025-11-30 16:06:17,316 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:06:17,334 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a058a7d8b615fd0b02930 HTTP/1.1" 502 150 2025-11-30 16:06:17,334 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a058a7d8b615fd0b02930'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:06:17,334 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a058a7d8b615fd0b02930 2025-11-30 16:06:17,335 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a058a7d8b615fd0b02930 HTTP/1.1" 502 150 2025-11-30 16:06:17,335 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a058a7d8b615fd0b02930'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:06:19,335 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a058a7d8b615fd0b02930 2025-11-30 16:06:19,337 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a058a7d8b615fd0b02930 HTTP/1.1" 502 150 2025-11-30 16:06:19,337 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a058a7d8b615fd0b02930'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:06:23,337 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a058a7d8b615fd0b02930 2025-11-30 16:06:23,338 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a058a7d8b615fd0b02930 HTTP/1.1" 502 150 2025-11-30 16:06:23,338 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a058a7d8b615fd0b02930'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:50320 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:06:31,338 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a058a7d8b615fd0b02930 2025-11-30 16:06:31,340 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a058a7d8b615fd0b02930 HTTP/1.1" 502 150 2025-11-30 16:06:31,341 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a058a7d8b615fd0b02930'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:06:34,445 - lava_callback - INFO - Received callback for node 692a58e2f8f845693da44c69 INFO: 172.19.0.1:49034 - "POST /node/692a58e2f8f845693da44c69 HTTP/1.0" 202 Accepted 2025-11-30 16:06:34,451 - lava_callback - INFO - Starting async job submit for node 692a58e2f8f845693da44c69 2025-11-30 16:06:34,468 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:06:34,477 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a58e2f8f845693da44c69 HTTP/1.1" 502 150 2025-11-30 16:06:34,478 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a58e2f8f845693da44c69'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:06:34,478 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a58e2f8f845693da44c69 2025-11-30 16:06:34,478 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a58e2f8f845693da44c69 HTTP/1.1" 502 150 2025-11-30 16:06:34,478 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a58e2f8f845693da44c69'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:06:36,479 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a58e2f8f845693da44c69 2025-11-30 16:06:36,480 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a58e2f8f845693da44c69 HTTP/1.1" 502 150 2025-11-30 16:06:36,480 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a58e2f8f845693da44c69'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:49038 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:06:40,480 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a58e2f8f845693da44c69 2025-11-30 16:06:40,481 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a58e2f8f845693da44c69 HTTP/1.1" 502 150 2025-11-30 16:06:40,482 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a58e2f8f845693da44c69'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:44686 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:06:47,341 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a058a7d8b615fd0b02930 2025-11-30 16:06:47,342 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a058a7d8b615fd0b02930 HTTP/1.1" 502 150 2025-11-30 16:06:47,342 - lava_callback - ERROR - Error processing callback for node 692a058a7d8b615fd0b02930: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692a058a7d8b615fd0b02930 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692a058a7d8b615fd0b02930 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692a058a7d8b615fd0b02930 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:06:48,482 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a58e2f8f845693da44c69 2025-11-30 16:06:48,483 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a58e2f8f845693da44c69 HTTP/1.1" 502 150 2025-11-30 16:06:48,483 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a58e2f8f845693da44c69'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:59668 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:07:04,483 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a58e2f8f845693da44c69 2025-11-30 16:07:04,485 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a58e2f8f845693da44c69 HTTP/1.1" 502 150 2025-11-30 16:07:04,485 - lava_callback - ERROR - Error processing callback for node 692a58e2f8f845693da44c69: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692a58e2f8f845693da44c69 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692a58e2f8f845693da44c69 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692a58e2f8f845693da44c69 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:40718 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:55908 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:55916 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:55922 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:56628 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:07:29,233 - lava_callback - INFO - Received callback for node 692acf40d481a0c6f8cb594a INFO: 172.19.0.1:56632 - "POST /node/692acf40d481a0c6f8cb594a HTTP/1.0" 202 Accepted 2025-11-30 16:07:29,240 - lava_callback - INFO - Starting async job submit for node 692acf40d481a0c6f8cb594a 2025-11-30 16:07:29,344 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:07:29,355 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf40d481a0c6f8cb594a HTTP/1.1" 502 150 2025-11-30 16:07:29,355 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf40d481a0c6f8cb594a'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:07:29,355 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf40d481a0c6f8cb594a 2025-11-30 16:07:29,356 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf40d481a0c6f8cb594a HTTP/1.1" 502 150 2025-11-30 16:07:29,356 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf40d481a0c6f8cb594a'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:07:31,357 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf40d481a0c6f8cb594a 2025-11-30 16:07:31,358 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf40d481a0c6f8cb594a HTTP/1.1" 502 150 2025-11-30 16:07:31,358 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf40d481a0c6f8cb594a'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:07:35,358 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf40d481a0c6f8cb594a 2025-11-30 16:07:35,363 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf40d481a0c6f8cb594a HTTP/1.1" 502 150 2025-11-30 16:07:35,363 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf40d481a0c6f8cb594a'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:46408 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:07:43,363 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf40d481a0c6f8cb594a 2025-11-30 16:07:43,364 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf40d481a0c6f8cb594a HTTP/1.1" 502 150 2025-11-30 16:07:43,364 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf40d481a0c6f8cb594a'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:58892 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:52662 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:07:59,365 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf40d481a0c6f8cb594a 2025-11-30 16:07:59,366 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf40d481a0c6f8cb594a HTTP/1.1" 502 150 2025-11-30 16:07:59,366 - lava_callback - ERROR - Error processing callback for node 692acf40d481a0c6f8cb594a: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf40d481a0c6f8cb594a (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf40d481a0c6f8cb594a (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf40d481a0c6f8cb594a (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:44172 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:51918 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:08:14,034 - lava_callback - INFO - Received callback for node 692a5e0ff8f845693da45097 INFO: 172.19.0.1:51930 - "POST /node/692a5e0ff8f845693da45097 HTTP/1.0" 202 Accepted 2025-11-30 16:08:14,040 - lava_callback - INFO - Starting async job submit for node 692a5e0ff8f845693da45097 2025-11-30 16:08:14,057 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:08:14,070 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a5e0ff8f845693da45097 HTTP/1.1" 502 150 2025-11-30 16:08:14,070 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a5e0ff8f845693da45097'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:08:14,071 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a5e0ff8f845693da45097 2025-11-30 16:08:14,071 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a5e0ff8f845693da45097 HTTP/1.1" 502 150 2025-11-30 16:08:14,071 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a5e0ff8f845693da45097'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:51934 - "GET /api/metrics HTTP/1.0" 200 OK 2025-11-30 16:08:16,072 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a5e0ff8f845693da45097 2025-11-30 16:08:16,073 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a5e0ff8f845693da45097 HTTP/1.1" 502 150 2025-11-30 16:08:16,073 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a5e0ff8f845693da45097'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:51938 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:08:20,074 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a5e0ff8f845693da45097 2025-11-30 16:08:20,075 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a5e0ff8f845693da45097 HTTP/1.1" 502 150 2025-11-30 16:08:20,075 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a5e0ff8f845693da45097'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:39544 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:08:28,075 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a5e0ff8f845693da45097 2025-11-30 16:08:28,076 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a5e0ff8f845693da45097 HTTP/1.1" 502 150 2025-11-30 16:08:28,077 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a5e0ff8f845693da45097'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:58934 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:08:44,077 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a5e0ff8f845693da45097 2025-11-30 16:08:44,078 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a5e0ff8f845693da45097 HTTP/1.1" 502 150 2025-11-30 16:08:44,078 - lava_callback - ERROR - Error processing callback for node 692a5e0ff8f845693da45097: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692a5e0ff8f845693da45097 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692a5e0ff8f845693da45097 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692a5e0ff8f845693da45097 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:55914 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:39860 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:48134 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:47140 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:09:12,722 - lava_callback - INFO - Received callback for node 692acf46d481a0c6f8cb5956 INFO: 172.19.0.1:47154 - "POST /node/692acf46d481a0c6f8cb5956 HTTP/1.0" 202 Accepted 2025-11-30 16:09:12,728 - lava_callback - INFO - Starting async job submit for node 692acf46d481a0c6f8cb5956 2025-11-30 16:09:12,754 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:09:12,783 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf46d481a0c6f8cb5956 HTTP/1.1" 502 150 2025-11-30 16:09:12,783 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf46d481a0c6f8cb5956'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:09:12,783 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf46d481a0c6f8cb5956 2025-11-30 16:09:12,784 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf46d481a0c6f8cb5956 HTTP/1.1" 502 150 2025-11-30 16:09:12,784 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf46d481a0c6f8cb5956'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:09:14,784 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf46d481a0c6f8cb5956 2025-11-30 16:09:14,785 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf46d481a0c6f8cb5956 HTTP/1.1" 502 150 2025-11-30 16:09:14,785 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf46d481a0c6f8cb5956'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:47156 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:47160 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:09:18,785 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf46d481a0c6f8cb5956 2025-11-30 16:09:18,787 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf46d481a0c6f8cb5956 HTTP/1.1" 502 150 2025-11-30 16:09:18,787 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf46d481a0c6f8cb5956'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:43370 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:09:26,787 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf46d481a0c6f8cb5956 2025-11-30 16:09:26,788 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf46d481a0c6f8cb5956 HTTP/1.1" 502 150 2025-11-30 16:09:26,788 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf46d481a0c6f8cb5956'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:55446 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:09:42,788 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf46d481a0c6f8cb5956 2025-11-30 16:09:42,789 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf46d481a0c6f8cb5956 HTTP/1.1" 502 150 2025-11-30 16:09:42,790 - lava_callback - ERROR - Error processing callback for node 692acf46d481a0c6f8cb5956: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf46d481a0c6f8cb5956 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf46d481a0c6f8cb5956 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf46d481a0c6f8cb5956 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:50482 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:60276 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:53712 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:43400 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:43404 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:43406 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:47832 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:10:32,873 - lava_callback - INFO - Received callback for node 692acf45d481a0c6f8cb5953 INFO: 172.19.0.1:33708 - "POST /node/692acf45d481a0c6f8cb5953 HTTP/1.0" 202 Accepted 2025-11-30 16:10:32,884 - lava_callback - INFO - Starting async job submit for node 692acf45d481a0c6f8cb5953 2025-11-30 16:10:32,994 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:10:33,026 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf45d481a0c6f8cb5953 HTTP/1.1" 502 150 2025-11-30 16:10:33,026 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf45d481a0c6f8cb5953'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:10:33,027 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf45d481a0c6f8cb5953 2025-11-30 16:10:33,027 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf45d481a0c6f8cb5953 HTTP/1.1" 502 150 2025-11-30 16:10:33,027 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf45d481a0c6f8cb5953'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:10:35,028 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf45d481a0c6f8cb5953 2025-11-30 16:10:35,029 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf45d481a0c6f8cb5953 HTTP/1.1" 502 150 2025-11-30 16:10:35,029 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf45d481a0c6f8cb5953'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:33724 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:10:39,029 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf45d481a0c6f8cb5953 2025-11-30 16:10:39,031 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf45d481a0c6f8cb5953 HTTP/1.1" 502 150 2025-11-30 16:10:39,031 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf45d481a0c6f8cb5953'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:40970 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:10:47,031 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf45d481a0c6f8cb5953 2025-11-30 16:10:47,032 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf45d481a0c6f8cb5953 HTTP/1.1" 502 150 2025-11-30 16:10:47,032 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf45d481a0c6f8cb5953'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:10:54,065 - lava_callback - INFO - Received callback for node 6929c5b0498c72996de1fcc1 INFO: 172.19.0.1:53342 - "POST /node/6929c5b0498c72996de1fcc1 HTTP/1.0" 202 Accepted 2025-11-30 16:10:54,087 - lava_callback - INFO - Starting async job submit for node 6929c5b0498c72996de1fcc1 2025-11-30 16:10:55,599 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:10:55,609 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c5b0498c72996de1fcc1 HTTP/1.1" 502 150 2025-11-30 16:10:55,609 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c5b0498c72996de1fcc1'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:10:55,609 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c5b0498c72996de1fcc1 2025-11-30 16:10:55,610 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c5b0498c72996de1fcc1 HTTP/1.1" 502 150 2025-11-30 16:10:55,610 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c5b0498c72996de1fcc1'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:53358 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:10:57,610 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c5b0498c72996de1fcc1 2025-11-30 16:10:57,611 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c5b0498c72996de1fcc1 HTTP/1.1" 502 150 2025-11-30 16:10:57,611 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c5b0498c72996de1fcc1'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:11:01,611 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c5b0498c72996de1fcc1 2025-11-30 16:11:01,613 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c5b0498c72996de1fcc1 HTTP/1.1" 502 150 2025-11-30 16:11:01,613 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c5b0498c72996de1fcc1'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:11:03,033 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf45d481a0c6f8cb5953 2025-11-30 16:11:03,034 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf45d481a0c6f8cb5953 HTTP/1.1" 502 150 2025-11-30 16:11:03,035 - lava_callback - ERROR - Error processing callback for node 692acf45d481a0c6f8cb5953: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf45d481a0c6f8cb5953 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf45d481a0c6f8cb5953 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf45d481a0c6f8cb5953 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:51418 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:11:09,613 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c5b0498c72996de1fcc1 2025-11-30 16:11:09,614 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c5b0498c72996de1fcc1 HTTP/1.1" 502 150 2025-11-30 16:11:09,614 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929c5b0498c72996de1fcc1'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:39610 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:39614 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:39628 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:11:25,614 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929c5b0498c72996de1fcc1 2025-11-30 16:11:25,616 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929c5b0498c72996de1fcc1 HTTP/1.1" 502 150 2025-11-30 16:11:25,616 - lava_callback - ERROR - Error processing callback for node 6929c5b0498c72996de1fcc1: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c5b0498c72996de1fcc1 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c5b0498c72996de1fcc1 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929c5b0498c72996de1fcc1 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:51750 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:11:29,119 - lava_callback - INFO - Received callback for node 692acf48d481a0c6f8cb5959 INFO: 172.19.0.1:51766 - "POST /node/692acf48d481a0c6f8cb5959 HTTP/1.0" 202 Accepted 2025-11-30 16:11:29,126 - lava_callback - INFO - Starting async job submit for node 692acf48d481a0c6f8cb5959 2025-11-30 16:11:29,156 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:11:29,165 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf48d481a0c6f8cb5959 HTTP/1.1" 502 150 2025-11-30 16:11:29,165 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf48d481a0c6f8cb5959'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:11:29,166 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf48d481a0c6f8cb5959 2025-11-30 16:11:29,166 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf48d481a0c6f8cb5959 HTTP/1.1" 502 150 2025-11-30 16:11:29,166 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf48d481a0c6f8cb5959'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:11:31,166 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf48d481a0c6f8cb5959 2025-11-30 16:11:31,168 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf48d481a0c6f8cb5959 HTTP/1.1" 502 150 2025-11-30 16:11:31,168 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf48d481a0c6f8cb5959'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:11:35,168 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf48d481a0c6f8cb5959 2025-11-30 16:11:35,169 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf48d481a0c6f8cb5959 HTTP/1.1" 502 150 2025-11-30 16:11:35,169 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf48d481a0c6f8cb5959'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:55566 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:11:43,170 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf48d481a0c6f8cb5959 2025-11-30 16:11:43,171 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf48d481a0c6f8cb5959 HTTP/1.1" 502 150 2025-11-30 16:11:43,171 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf48d481a0c6f8cb5959'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:52894 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:57260 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:11:59,171 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf48d481a0c6f8cb5959 2025-11-30 16:11:59,172 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf48d481a0c6f8cb5959 HTTP/1.1" 502 150 2025-11-30 16:11:59,173 - lava_callback - ERROR - Error processing callback for node 692acf48d481a0c6f8cb5959: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf48d481a0c6f8cb5959 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf48d481a0c6f8cb5959 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf48d481a0c6f8cb5959 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:52750 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:40988 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:40998 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:41008 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:48280 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:12:32,727 - lava_callback - INFO - Received callback for node 6929ccfc7d8b615fd0b00d19 INFO: 172.19.0.1:55404 - "POST /node/6929ccfc7d8b615fd0b00d19 HTTP/1.0" 202 Accepted 2025-11-30 16:12:32,738 - lava_callback - INFO - Starting async job submit for node 6929ccfc7d8b615fd0b00d19 2025-11-30 16:12:32,777 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:12:32,790 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929ccfc7d8b615fd0b00d19 HTTP/1.1" 502 150 2025-11-30 16:12:32,790 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929ccfc7d8b615fd0b00d19'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:12:32,790 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929ccfc7d8b615fd0b00d19 2025-11-30 16:12:32,791 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929ccfc7d8b615fd0b00d19 HTTP/1.1" 502 150 2025-11-30 16:12:32,791 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929ccfc7d8b615fd0b00d19'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:12:34,791 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929ccfc7d8b615fd0b00d19 2025-11-30 16:12:34,793 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929ccfc7d8b615fd0b00d19 HTTP/1.1" 502 150 2025-11-30 16:12:34,793 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929ccfc7d8b615fd0b00d19'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:55416 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:12:38,793 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929ccfc7d8b615fd0b00d19 2025-11-30 16:12:38,794 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929ccfc7d8b615fd0b00d19 HTTP/1.1" 502 150 2025-11-30 16:12:38,794 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929ccfc7d8b615fd0b00d19'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:45198 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:12:46,794 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929ccfc7d8b615fd0b00d19 2025-11-30 16:12:46,796 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929ccfc7d8b615fd0b00d19 HTTP/1.1" 502 150 2025-11-30 16:12:46,796 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929ccfc7d8b615fd0b00d19'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:12:53,579 - lava_callback - INFO - Received callback for node 692acf41d481a0c6f8cb594d INFO: 172.19.0.1:39694 - "POST /node/692acf41d481a0c6f8cb594d HTTP/1.0" 202 Accepted 2025-11-30 16:12:53,592 - lava_callback - INFO - Starting async job submit for node 692acf41d481a0c6f8cb594d 2025-11-30 16:12:53,979 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:12:53,993 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf41d481a0c6f8cb594d HTTP/1.1" 502 150 2025-11-30 16:12:53,993 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf41d481a0c6f8cb594d'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:12:53,993 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf41d481a0c6f8cb594d 2025-11-30 16:12:53,994 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf41d481a0c6f8cb594d HTTP/1.1" 502 150 2025-11-30 16:12:53,994 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf41d481a0c6f8cb594d'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:12:55,994 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf41d481a0c6f8cb594d 2025-11-30 16:12:55,995 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf41d481a0c6f8cb594d HTTP/1.1" 502 150 2025-11-30 16:12:55,995 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf41d481a0c6f8cb594d'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:39710 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:12:59,996 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf41d481a0c6f8cb594d 2025-11-30 16:12:59,997 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf41d481a0c6f8cb594d HTTP/1.1" 502 150 2025-11-30 16:12:59,997 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf41d481a0c6f8cb594d'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:13:02,796 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929ccfc7d8b615fd0b00d19 2025-11-30 16:13:02,797 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929ccfc7d8b615fd0b00d19 HTTP/1.1" 502 150 2025-11-30 16:13:02,797 - lava_callback - ERROR - Error processing callback for node 6929ccfc7d8b615fd0b00d19: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929ccfc7d8b615fd0b00d19 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929ccfc7d8b615fd0b00d19 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929ccfc7d8b615fd0b00d19 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:44176 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:13:07,997 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf41d481a0c6f8cb594d 2025-11-30 16:13:07,999 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf41d481a0c6f8cb594d HTTP/1.1" 502 150 2025-11-30 16:13:07,999 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf41d481a0c6f8cb594d'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:35366 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:13:13,316 - lava_callback - INFO - Received callback for node 692acf4ad481a0c6f8cb595c INFO: 172.19.0.1:35376 - "POST /node/692acf4ad481a0c6f8cb595c HTTP/1.0" 202 Accepted 2025-11-30 16:13:13,323 - lava_callback - INFO - Starting async job submit for node 692acf4ad481a0c6f8cb595c 2025-11-30 16:13:13,353 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:13:13,364 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf4ad481a0c6f8cb595c HTTP/1.1" 502 150 2025-11-30 16:13:13,365 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf4ad481a0c6f8cb595c'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:13:13,365 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf4ad481a0c6f8cb595c 2025-11-30 16:13:13,366 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf4ad481a0c6f8cb595c HTTP/1.1" 502 150 2025-11-30 16:13:13,366 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf4ad481a0c6f8cb595c'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:35382 - "GET /api/metrics HTTP/1.0" 200 OK 2025-11-30 16:13:15,366 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf4ad481a0c6f8cb595c 2025-11-30 16:13:15,367 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf4ad481a0c6f8cb595c HTTP/1.1" 502 150 2025-11-30 16:13:15,367 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf4ad481a0c6f8cb595c'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:35388 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:13:19,369 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf4ad481a0c6f8cb595c 2025-11-30 16:13:19,371 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf4ad481a0c6f8cb595c HTTP/1.1" 502 150 2025-11-30 16:13:19,371 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf4ad481a0c6f8cb595c'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:13:23,999 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf41d481a0c6f8cb594d 2025-11-30 16:13:24,000 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf41d481a0c6f8cb594d HTTP/1.1" 502 150 2025-11-30 16:13:24,000 - lava_callback - ERROR - Error processing callback for node 692acf41d481a0c6f8cb594d: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf41d481a0c6f8cb594d (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf41d481a0c6f8cb594d (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf41d481a0c6f8cb594d (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:50034 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:13:27,371 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf4ad481a0c6f8cb595c 2025-11-30 16:13:27,372 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf4ad481a0c6f8cb595c HTTP/1.1" 502 150 2025-11-30 16:13:27,372 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf4ad481a0c6f8cb595c'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:56018 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:13:43,373 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf4ad481a0c6f8cb595c 2025-11-30 16:13:43,374 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf4ad481a0c6f8cb595c HTTP/1.1" 502 150 2025-11-30 16:13:43,374 - lava_callback - ERROR - Error processing callback for node 692acf4ad481a0c6f8cb595c: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf4ad481a0c6f8cb595c (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf4ad481a0c6f8cb595c (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf4ad481a0c6f8cb595c (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:57778 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:37476 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:38564 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:41176 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:41182 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:41188 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:38288 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:43562 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:50768 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:41830 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:57956 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:43516 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:43528 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:43530 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:58572 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:34122 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:40008 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:15:47,805 - lava_callback - INFO - Received callback for node 6929ca9e7d8b615fd0b00c3f INFO: 172.19.0.1:40010 - "POST /node/6929ca9e7d8b615fd0b00c3f HTTP/1.0" 202 Accepted 2025-11-30 16:15:47,811 - lava_callback - INFO - Starting async job submit for node 6929ca9e7d8b615fd0b00c3f 2025-11-30 16:15:47,811 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:15:47,821 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929ca9e7d8b615fd0b00c3f HTTP/1.1" 502 150 2025-11-30 16:15:47,821 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929ca9e7d8b615fd0b00c3f'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:15:47,821 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929ca9e7d8b615fd0b00c3f 2025-11-30 16:15:47,822 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929ca9e7d8b615fd0b00c3f HTTP/1.1" 502 150 2025-11-30 16:15:47,822 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929ca9e7d8b615fd0b00c3f'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:15:49,822 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929ca9e7d8b615fd0b00c3f 2025-11-30 16:15:49,824 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929ca9e7d8b615fd0b00c3f HTTP/1.1" 502 150 2025-11-30 16:15:49,824 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929ca9e7d8b615fd0b00c3f'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:15:53,824 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929ca9e7d8b615fd0b00c3f 2025-11-30 16:15:53,825 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929ca9e7d8b615fd0b00c3f HTTP/1.1" 502 150 2025-11-30 16:15:53,825 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929ca9e7d8b615fd0b00c3f'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:45654 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:16:01,825 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929ca9e7d8b615fd0b00c3f 2025-11-30 16:16:01,827 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929ca9e7d8b615fd0b00c3f HTTP/1.1" 502 150 2025-11-30 16:16:01,827 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929ca9e7d8b615fd0b00c3f'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:53018 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:40124 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:40132 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:40134 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:16:17,599 - lava_callback - INFO - Received callback for node 692a58a3f8f845693da44c5d INFO: 172.19.0.1:40150 - "POST /node/692a58a3f8f845693da44c5d HTTP/1.0" 202 Accepted 2025-11-30 16:16:17,605 - lava_callback - INFO - Starting async job submit for node 692a58a3f8f845693da44c5d 2025-11-30 16:16:17,636 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:16:17,646 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a58a3f8f845693da44c5d HTTP/1.1" 502 150 2025-11-30 16:16:17,646 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a58a3f8f845693da44c5d'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:16:17,646 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a58a3f8f845693da44c5d 2025-11-30 16:16:17,647 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a58a3f8f845693da44c5d HTTP/1.1" 502 150 2025-11-30 16:16:17,647 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a58a3f8f845693da44c5d'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:16:17,827 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929ca9e7d8b615fd0b00c3f 2025-11-30 16:16:17,828 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929ca9e7d8b615fd0b00c3f HTTP/1.1" 502 150 2025-11-30 16:16:17,828 - lava_callback - ERROR - Error processing callback for node 6929ca9e7d8b615fd0b00c3f: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929ca9e7d8b615fd0b00c3f (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929ca9e7d8b615fd0b00c3f (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929ca9e7d8b615fd0b00c3f (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:16:19,647 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a58a3f8f845693da44c5d 2025-11-30 16:16:19,648 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a58a3f8f845693da44c5d HTTP/1.1" 502 150 2025-11-30 16:16:19,648 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a58a3f8f845693da44c5d'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:16:23,648 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a58a3f8f845693da44c5d 2025-11-30 16:16:23,649 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a58a3f8f845693da44c5d HTTP/1.1" 502 150 2025-11-30 16:16:23,650 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a58a3f8f845693da44c5d'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:38754 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:16:27,802 - lava_callback - INFO - Received callback for node 6929cad57d8b615fd0b00c47 INFO: 172.19.0.1:38756 - "POST /node/6929cad57d8b615fd0b00c47 HTTP/1.0" 202 Accepted 2025-11-30 16:16:27,808 - lava_callback - INFO - Starting async job submit for node 6929cad57d8b615fd0b00c47 2025-11-30 16:16:27,809 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:16:27,831 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cad57d8b615fd0b00c47 HTTP/1.1" 502 150 2025-11-30 16:16:27,831 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cad57d8b615fd0b00c47'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:16:27,831 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cad57d8b615fd0b00c47 2025-11-30 16:16:27,832 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cad57d8b615fd0b00c47 HTTP/1.1" 502 150 2025-11-30 16:16:27,832 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cad57d8b615fd0b00c47'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:16:28,449 - lava_callback - INFO - Received callback for node 6929cad87d8b615fd0b00c4a INFO: 172.19.0.1:38764 - "POST /node/6929cad87d8b615fd0b00c4a HTTP/1.0" 202 Accepted 2025-11-30 16:16:28,457 - lava_callback - INFO - Starting async job submit for node 6929cad87d8b615fd0b00c4a 2025-11-30 16:16:28,458 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:16:28,472 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cad87d8b615fd0b00c4a HTTP/1.1" 502 150 2025-11-30 16:16:28,472 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cad87d8b615fd0b00c4a'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:16:28,473 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cad87d8b615fd0b00c4a 2025-11-30 16:16:28,473 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cad87d8b615fd0b00c4a HTTP/1.1" 502 150 2025-11-30 16:16:28,473 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cad87d8b615fd0b00c4a'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:16:29,832 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cad57d8b615fd0b00c47 2025-11-30 16:16:29,833 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cad57d8b615fd0b00c47 HTTP/1.1" 502 150 2025-11-30 16:16:29,833 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cad57d8b615fd0b00c47'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:16:30,474 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cad87d8b615fd0b00c4a 2025-11-30 16:16:30,475 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cad87d8b615fd0b00c4a HTTP/1.1" 502 150 2025-11-30 16:16:30,475 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cad87d8b615fd0b00c4a'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:16:31,650 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a58a3f8f845693da44c5d 2025-11-30 16:16:31,651 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a58a3f8f845693da44c5d HTTP/1.1" 502 150 2025-11-30 16:16:31,651 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a58a3f8f845693da44c5d'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:16:33,834 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cad57d8b615fd0b00c47 2025-11-30 16:16:33,835 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cad57d8b615fd0b00c47 HTTP/1.1" 502 150 2025-11-30 16:16:33,835 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cad57d8b615fd0b00c47'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:16:34,475 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cad87d8b615fd0b00c4a 2025-11-30 16:16:34,476 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cad87d8b615fd0b00c4a HTTP/1.1" 502 150 2025-11-30 16:16:34,476 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cad87d8b615fd0b00c4a'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:45260 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:16:41,835 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cad57d8b615fd0b00c47 2025-11-30 16:16:41,836 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cad57d8b615fd0b00c47 HTTP/1.1" 502 150 2025-11-30 16:16:41,836 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cad57d8b615fd0b00c47'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:16:42,477 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cad87d8b615fd0b00c4a 2025-11-30 16:16:42,477 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cad87d8b615fd0b00c4a HTTP/1.1" 502 150 2025-11-30 16:16:42,477 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cad87d8b615fd0b00c4a'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:35504 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:16:47,651 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a58a3f8f845693da44c5d 2025-11-30 16:16:47,652 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a58a3f8f845693da44c5d HTTP/1.1" 502 150 2025-11-30 16:16:47,653 - lava_callback - ERROR - Error processing callback for node 692a58a3f8f845693da44c5d: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692a58a3f8f845693da44c5d (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692a58a3f8f845693da44c5d (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692a58a3f8f845693da44c5d (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:16:47,854 - lava_callback - INFO - Received callback for node 6929cadb7d8b615fd0b00c4d INFO: 172.19.0.1:35510 - "POST /node/6929cadb7d8b615fd0b00c4d HTTP/1.0" 202 Accepted 2025-11-30 16:16:47,860 - lava_callback - INFO - Starting async job submit for node 6929cadb7d8b615fd0b00c4d 2025-11-30 16:16:47,860 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:16:47,870 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cadb7d8b615fd0b00c4d HTTP/1.1" 502 150 2025-11-30 16:16:47,870 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cadb7d8b615fd0b00c4d'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:16:47,870 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cadb7d8b615fd0b00c4d 2025-11-30 16:16:47,871 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cadb7d8b615fd0b00c4d HTTP/1.1" 502 150 2025-11-30 16:16:47,871 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cadb7d8b615fd0b00c4d'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:16:49,871 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cadb7d8b615fd0b00c4d 2025-11-30 16:16:49,872 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cadb7d8b615fd0b00c4d HTTP/1.1" 502 150 2025-11-30 16:16:49,872 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cadb7d8b615fd0b00c4d'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:16:53,873 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cadb7d8b615fd0b00c4d 2025-11-30 16:16:53,874 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cadb7d8b615fd0b00c4d HTTP/1.1" 502 150 2025-11-30 16:16:53,874 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cadb7d8b615fd0b00c4d'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:34010 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:16:57,836 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cad57d8b615fd0b00c47 2025-11-30 16:16:57,838 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cad57d8b615fd0b00c47 HTTP/1.1" 502 150 2025-11-30 16:16:57,838 - lava_callback - ERROR - Error processing callback for node 6929cad57d8b615fd0b00c47: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cad57d8b615fd0b00c47 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cad57d8b615fd0b00c47 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cad57d8b615fd0b00c47 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:16:58,478 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cad87d8b615fd0b00c4a 2025-11-30 16:16:58,479 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cad87d8b615fd0b00c4a HTTP/1.1" 502 150 2025-11-30 16:16:58,479 - lava_callback - ERROR - Error processing callback for node 6929cad87d8b615fd0b00c4a: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cad87d8b615fd0b00c4a (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cad87d8b615fd0b00c4a (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cad87d8b615fd0b00c4a (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:17:01,874 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cadb7d8b615fd0b00c4d 2025-11-30 16:17:01,875 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cadb7d8b615fd0b00c4d HTTP/1.1" 502 150 2025-11-30 16:17:01,875 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cadb7d8b615fd0b00c4d'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:45068 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:50638 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:50654 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:50660 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:17:17,875 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cadb7d8b615fd0b00c4d 2025-11-30 16:17:17,876 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cadb7d8b615fd0b00c4d HTTP/1.1" 502 150 2025-11-30 16:17:17,877 - lava_callback - ERROR - Error processing callback for node 6929cadb7d8b615fd0b00c4d: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cadb7d8b615fd0b00c4d (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cadb7d8b615fd0b00c4d (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cadb7d8b615fd0b00c4d (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:48662 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:17:27,892 - lava_callback - INFO - Received callback for node 6929cb127d8b615fd0b00c73 INFO: 172.19.0.1:48676 - "POST /node/6929cb127d8b615fd0b00c73 HTTP/1.0" 202 Accepted 2025-11-30 16:17:27,897 - lava_callback - INFO - Starting async job submit for node 6929cb127d8b615fd0b00c73 2025-11-30 16:17:27,898 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:17:27,922 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb127d8b615fd0b00c73 HTTP/1.1" 502 150 2025-11-30 16:17:27,922 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb127d8b615fd0b00c73'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:27,922 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb127d8b615fd0b00c73 2025-11-30 16:17:27,923 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb127d8b615fd0b00c73 HTTP/1.1" 502 150 2025-11-30 16:17:27,923 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb127d8b615fd0b00c73'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:28,541 - lava_callback - INFO - Received callback for node 6929cb157d8b615fd0b00c76 INFO: 172.19.0.1:48684 - "POST /node/6929cb157d8b615fd0b00c76 HTTP/1.0" 202 Accepted 2025-11-30 16:17:28,547 - lava_callback - INFO - Starting async job submit for node 6929cb157d8b615fd0b00c76 2025-11-30 16:17:28,548 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:17:28,557 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb157d8b615fd0b00c76 HTTP/1.1" 502 150 2025-11-30 16:17:28,557 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb157d8b615fd0b00c76'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:28,558 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb157d8b615fd0b00c76 2025-11-30 16:17:28,558 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb157d8b615fd0b00c76 HTTP/1.1" 502 150 2025-11-30 16:17:28,558 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb157d8b615fd0b00c76'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:29,923 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb127d8b615fd0b00c73 2025-11-30 16:17:29,924 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb127d8b615fd0b00c73 HTTP/1.1" 502 150 2025-11-30 16:17:29,924 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb127d8b615fd0b00c73'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:30,559 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb157d8b615fd0b00c76 2025-11-30 16:17:30,560 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb157d8b615fd0b00c76 HTTP/1.1" 502 150 2025-11-30 16:17:30,560 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb157d8b615fd0b00c76'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:33,029 - lava_callback - INFO - Received callback for node 692acf4fd481a0c6f8cb5965 INFO: 172.19.0.1:51986 - "POST /node/692acf4fd481a0c6f8cb5965 HTTP/1.0" 202 Accepted 2025-11-30 16:17:33,035 - lava_callback - INFO - Starting async job submit for node 692acf4fd481a0c6f8cb5965 2025-11-30 16:17:33,077 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:17:33,087 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf4fd481a0c6f8cb5965 HTTP/1.1" 502 150 2025-11-30 16:17:33,087 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf4fd481a0c6f8cb5965'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:33,087 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf4fd481a0c6f8cb5965 2025-11-30 16:17:33,088 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf4fd481a0c6f8cb5965 HTTP/1.1" 502 150 2025-11-30 16:17:33,088 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf4fd481a0c6f8cb5965'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:33,925 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb127d8b615fd0b00c73 2025-11-30 16:17:33,926 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb127d8b615fd0b00c73 HTTP/1.1" 502 150 2025-11-30 16:17:33,926 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb127d8b615fd0b00c73'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:34,560 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb157d8b615fd0b00c76 2025-11-30 16:17:34,561 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb157d8b615fd0b00c76 HTTP/1.1" 502 150 2025-11-30 16:17:34,561 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb157d8b615fd0b00c76'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:35,088 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf4fd481a0c6f8cb5965 2025-11-30 16:17:35,089 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf4fd481a0c6f8cb5965 HTTP/1.1" 502 150 2025-11-30 16:17:35,089 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf4fd481a0c6f8cb5965'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:51998 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:17:39,090 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf4fd481a0c6f8cb5965 2025-11-30 16:17:39,092 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf4fd481a0c6f8cb5965 HTTP/1.1" 502 150 2025-11-30 16:17:39,092 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf4fd481a0c6f8cb5965'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:41,926 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb127d8b615fd0b00c73 2025-11-30 16:17:41,927 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb127d8b615fd0b00c73 HTTP/1.1" 502 150 2025-11-30 16:17:41,928 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb127d8b615fd0b00c73'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:42,561 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb157d8b615fd0b00c76 2025-11-30 16:17:42,563 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb157d8b615fd0b00c76 HTTP/1.1" 502 150 2025-11-30 16:17:42,563 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb157d8b615fd0b00c76'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:54600 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:17:47,092 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf4fd481a0c6f8cb5965 2025-11-30 16:17:47,093 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf4fd481a0c6f8cb5965 HTTP/1.1" 502 150 2025-11-30 16:17:47,093 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf4fd481a0c6f8cb5965'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:47,902 - lava_callback - INFO - Received callback for node 6929cb187d8b615fd0b00c79 INFO: 172.19.0.1:54604 - "POST /node/6929cb187d8b615fd0b00c79 HTTP/1.0" 202 Accepted 2025-11-30 16:17:47,907 - lava_callback - INFO - Starting async job submit for node 6929cb187d8b615fd0b00c79 2025-11-30 16:17:47,908 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:17:47,940 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb187d8b615fd0b00c79 HTTP/1.1" 502 150 2025-11-30 16:17:47,940 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb187d8b615fd0b00c79'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:47,941 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb187d8b615fd0b00c79 2025-11-30 16:17:47,941 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb187d8b615fd0b00c79 HTTP/1.1" 502 150 2025-11-30 16:17:47,941 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb187d8b615fd0b00c79'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:48,549 - lava_callback - INFO - Received callback for node 6929cb1b7d8b615fd0b00c7c INFO: 172.19.0.1:54610 - "POST /node/6929cb1b7d8b615fd0b00c7c HTTP/1.0" 202 Accepted 2025-11-30 16:17:48,555 - lava_callback - INFO - Starting async job submit for node 6929cb1b7d8b615fd0b00c7c 2025-11-30 16:17:48,556 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:17:48,565 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb1b7d8b615fd0b00c7c HTTP/1.1" 502 150 2025-11-30 16:17:48,565 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb1b7d8b615fd0b00c7c'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:48,565 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb1b7d8b615fd0b00c7c 2025-11-30 16:17:48,566 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb1b7d8b615fd0b00c7c HTTP/1.1" 502 150 2025-11-30 16:17:48,566 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb1b7d8b615fd0b00c7c'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:49,216 - lava_callback - INFO - Received callback for node 6929cb1e7d8b615fd0b00c83 INFO: 172.19.0.1:54626 - "POST /node/6929cb1e7d8b615fd0b00c83 HTTP/1.0" 202 Accepted 2025-11-30 16:17:49,222 - lava_callback - INFO - Starting async job submit for node 6929cb1e7d8b615fd0b00c83 2025-11-30 16:17:49,223 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:17:49,233 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb1e7d8b615fd0b00c83 HTTP/1.1" 502 150 2025-11-30 16:17:49,233 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb1e7d8b615fd0b00c83'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:49,233 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb1e7d8b615fd0b00c83 2025-11-30 16:17:49,234 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb1e7d8b615fd0b00c83 HTTP/1.1" 502 150 2025-11-30 16:17:49,234 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb1e7d8b615fd0b00c83'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:49,941 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb187d8b615fd0b00c79 2025-11-30 16:17:49,943 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb187d8b615fd0b00c79 HTTP/1.1" 502 150 2025-11-30 16:17:49,943 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb187d8b615fd0b00c79'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:50,566 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb1b7d8b615fd0b00c7c 2025-11-30 16:17:50,568 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb1b7d8b615fd0b00c7c HTTP/1.1" 502 150 2025-11-30 16:17:50,568 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb1b7d8b615fd0b00c7c'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:51,234 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb1e7d8b615fd0b00c83 2025-11-30 16:17:51,235 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb1e7d8b615fd0b00c83 HTTP/1.1" 502 150 2025-11-30 16:17:51,235 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb1e7d8b615fd0b00c83'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:53,943 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb187d8b615fd0b00c79 2025-11-30 16:17:53,944 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb187d8b615fd0b00c79 HTTP/1.1" 502 150 2025-11-30 16:17:53,945 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb187d8b615fd0b00c79'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:54,568 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb1b7d8b615fd0b00c7c 2025-11-30 16:17:54,569 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb1b7d8b615fd0b00c7c HTTP/1.1" 502 150 2025-11-30 16:17:54,569 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb1b7d8b615fd0b00c7c'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:54,841 - lava_callback - INFO - Received callback for node 692a6f32f8f845693da4568a INFO: 172.19.0.1:54004 - "POST /node/692a6f32f8f845693da4568a HTTP/1.0" 202 Accepted 2025-11-30 16:17:54,847 - lava_callback - INFO - Starting async job submit for node 692a6f32f8f845693da4568a 2025-11-30 16:17:54,864 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:17:54,874 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a6f32f8f845693da4568a HTTP/1.1" 502 150 2025-11-30 16:17:54,874 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a6f32f8f845693da4568a'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:54,874 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a6f32f8f845693da4568a 2025-11-30 16:17:54,875 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a6f32f8f845693da4568a HTTP/1.1" 502 150 2025-11-30 16:17:54,875 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a6f32f8f845693da4568a'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:55,235 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb1e7d8b615fd0b00c83 2025-11-30 16:17:55,237 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb1e7d8b615fd0b00c83 HTTP/1.1" 502 150 2025-11-30 16:17:55,237 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb1e7d8b615fd0b00c83'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:54016 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:17:56,875 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a6f32f8f845693da4568a 2025-11-30 16:17:56,876 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a6f32f8f845693da4568a HTTP/1.1" 502 150 2025-11-30 16:17:56,877 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a6f32f8f845693da4568a'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:57,928 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb127d8b615fd0b00c73 2025-11-30 16:17:57,929 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb127d8b615fd0b00c73 HTTP/1.1" 502 150 2025-11-30 16:17:57,929 - lava_callback - ERROR - Error processing callback for node 6929cb127d8b615fd0b00c73: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cb127d8b615fd0b00c73 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cb127d8b615fd0b00c73 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cb127d8b615fd0b00c73 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:17:58,006 - lava_callback - INFO - Received callback for node 6929cb217d8b615fd0b00c86 INFO: 172.19.0.1:54032 - "POST /node/6929cb217d8b615fd0b00c86 HTTP/1.0" 202 Accepted 2025-11-30 16:17:58,017 - lava_callback - INFO - Starting async job submit for node 6929cb217d8b615fd0b00c86 2025-11-30 16:17:58,022 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:17:58,040 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb217d8b615fd0b00c86 HTTP/1.1" 502 150 2025-11-30 16:17:58,040 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb217d8b615fd0b00c86'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:58,040 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb217d8b615fd0b00c86 2025-11-30 16:17:58,041 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb217d8b615fd0b00c86 HTTP/1.1" 502 150 2025-11-30 16:17:58,041 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb217d8b615fd0b00c86'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:58,563 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb157d8b615fd0b00c76 2025-11-30 16:17:58,564 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb157d8b615fd0b00c76 HTTP/1.1" 502 150 2025-11-30 16:17:58,565 - lava_callback - ERROR - Error processing callback for node 6929cb157d8b615fd0b00c76: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cb157d8b615fd0b00c76 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cb157d8b615fd0b00c76 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cb157d8b615fd0b00c76 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:17:58,629 - lava_callback - INFO - Received callback for node 6929cb257d8b615fd0b00c89 INFO: 172.19.0.1:54040 - "POST /node/6929cb257d8b615fd0b00c89 HTTP/1.0" 202 Accepted 2025-11-30 16:17:58,638 - lava_callback - INFO - Starting async job submit for node 6929cb257d8b615fd0b00c89 2025-11-30 16:17:58,640 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:17:58,677 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb257d8b615fd0b00c89 HTTP/1.1" 502 150 2025-11-30 16:17:58,677 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb257d8b615fd0b00c89'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:17:58,677 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb257d8b615fd0b00c89 2025-11-30 16:17:58,678 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb257d8b615fd0b00c89 HTTP/1.1" 502 150 2025-11-30 16:17:58,678 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb257d8b615fd0b00c89'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:18:00,041 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb217d8b615fd0b00c86 2025-11-30 16:18:00,043 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb217d8b615fd0b00c86 HTTP/1.1" 502 150 2025-11-30 16:18:00,043 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb217d8b615fd0b00c86'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:18:00,678 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb257d8b615fd0b00c89 2025-11-30 16:18:00,680 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb257d8b615fd0b00c89 HTTP/1.1" 502 150 2025-11-30 16:18:00,680 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb257d8b615fd0b00c89'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:18:00,877 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a6f32f8f845693da4568a 2025-11-30 16:18:00,878 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a6f32f8f845693da4568a HTTP/1.1" 502 150 2025-11-30 16:18:00,878 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a6f32f8f845693da4568a'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:18:01,945 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb187d8b615fd0b00c79 2025-11-30 16:18:01,946 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb187d8b615fd0b00c79 HTTP/1.1" 502 150 2025-11-30 16:18:01,946 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb187d8b615fd0b00c79'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:18:02,569 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb1b7d8b615fd0b00c7c 2025-11-30 16:18:02,571 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb1b7d8b615fd0b00c7c HTTP/1.1" 502 150 2025-11-30 16:18:02,571 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb1b7d8b615fd0b00c7c'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:18:03,094 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf4fd481a0c6f8cb5965 2025-11-30 16:18:03,098 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf4fd481a0c6f8cb5965 HTTP/1.1" 502 150 2025-11-30 16:18:03,099 - lava_callback - ERROR - Error processing callback for node 692acf4fd481a0c6f8cb5965: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf4fd481a0c6f8cb5965 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf4fd481a0c6f8cb5965 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf4fd481a0c6f8cb5965 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:18:03,237 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb1e7d8b615fd0b00c83 2025-11-30 16:18:03,238 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb1e7d8b615fd0b00c83 HTTP/1.1" 502 150 2025-11-30 16:18:03,238 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb1e7d8b615fd0b00c83'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:18:04,043 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb217d8b615fd0b00c86 2025-11-30 16:18:04,044 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb217d8b615fd0b00c86 HTTP/1.1" 502 150 2025-11-30 16:18:04,044 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb217d8b615fd0b00c86'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:18:04,680 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb257d8b615fd0b00c89 2025-11-30 16:18:04,681 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb257d8b615fd0b00c89 HTTP/1.1" 502 150 2025-11-30 16:18:04,681 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb257d8b615fd0b00c89'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:41472 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:18:08,878 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a6f32f8f845693da4568a 2025-11-30 16:18:08,880 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a6f32f8f845693da4568a HTTP/1.1" 502 150 2025-11-30 16:18:08,880 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a6f32f8f845693da4568a'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:18:09,415 - lava_callback - INFO - Received callback for node 692acf4dd481a0c6f8cb5962 INFO: 172.19.0.1:41480 - "POST /node/692acf4dd481a0c6f8cb5962 HTTP/1.0" 202 Accepted 2025-11-30 16:18:09,422 - lava_callback - INFO - Starting async job submit for node 692acf4dd481a0c6f8cb5962 2025-11-30 16:18:09,454 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:18:09,464 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf4dd481a0c6f8cb5962 HTTP/1.1" 502 150 2025-11-30 16:18:09,464 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf4dd481a0c6f8cb5962'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:18:09,464 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf4dd481a0c6f8cb5962 2025-11-30 16:18:09,465 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf4dd481a0c6f8cb5962 HTTP/1.1" 502 150 2025-11-30 16:18:09,465 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf4dd481a0c6f8cb5962'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:18:11,465 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf4dd481a0c6f8cb5962 2025-11-30 16:18:11,466 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf4dd481a0c6f8cb5962 HTTP/1.1" 502 150 2025-11-30 16:18:11,467 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf4dd481a0c6f8cb5962'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:18:12,045 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb217d8b615fd0b00c86 2025-11-30 16:18:12,046 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb217d8b615fd0b00c86 HTTP/1.1" 502 150 2025-11-30 16:18:12,046 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb217d8b615fd0b00c86'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:18:12,681 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb257d8b615fd0b00c89 2025-11-30 16:18:12,683 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb257d8b615fd0b00c89 HTTP/1.1" 502 150 2025-11-30 16:18:12,683 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cb257d8b615fd0b00c89'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:33514 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:33524 - "GET /api/metrics HTTP/1.0" 200 OK 2025-11-30 16:18:15,467 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf4dd481a0c6f8cb5962 2025-11-30 16:18:15,468 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf4dd481a0c6f8cb5962 HTTP/1.1" 502 150 2025-11-30 16:18:15,469 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf4dd481a0c6f8cb5962'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:33532 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:18:17,946 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb187d8b615fd0b00c79 2025-11-30 16:18:17,948 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb187d8b615fd0b00c79 HTTP/1.1" 502 150 2025-11-30 16:18:17,948 - lava_callback - ERROR - Error processing callback for node 6929cb187d8b615fd0b00c79: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cb187d8b615fd0b00c79 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cb187d8b615fd0b00c79 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cb187d8b615fd0b00c79 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:18:18,571 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb1b7d8b615fd0b00c7c 2025-11-30 16:18:18,572 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb1b7d8b615fd0b00c7c HTTP/1.1" 502 150 2025-11-30 16:18:18,572 - lava_callback - ERROR - Error processing callback for node 6929cb1b7d8b615fd0b00c7c: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cb1b7d8b615fd0b00c7c (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cb1b7d8b615fd0b00c7c (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cb1b7d8b615fd0b00c7c (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:18:19,238 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb1e7d8b615fd0b00c83 2025-11-30 16:18:19,240 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb1e7d8b615fd0b00c83 HTTP/1.1" 502 150 2025-11-30 16:18:19,240 - lava_callback - ERROR - Error processing callback for node 6929cb1e7d8b615fd0b00c83: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cb1e7d8b615fd0b00c83 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cb1e7d8b615fd0b00c83 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cb1e7d8b615fd0b00c83 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:18:23,469 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf4dd481a0c6f8cb5962 2025-11-30 16:18:23,471 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf4dd481a0c6f8cb5962 HTTP/1.1" 502 150 2025-11-30 16:18:23,471 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf4dd481a0c6f8cb5962'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:18:24,880 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a6f32f8f845693da4568a 2025-11-30 16:18:24,881 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a6f32f8f845693da4568a HTTP/1.1" 502 150 2025-11-30 16:18:24,881 - lava_callback - ERROR - Error processing callback for node 692a6f32f8f845693da4568a: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692a6f32f8f845693da4568a (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692a6f32f8f845693da4568a (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692a6f32f8f845693da4568a (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:39214 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:18:28,046 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb217d8b615fd0b00c86 2025-11-30 16:18:28,048 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb217d8b615fd0b00c86 HTTP/1.1" 502 150 2025-11-30 16:18:28,048 - lava_callback - ERROR - Error processing callback for node 6929cb217d8b615fd0b00c86: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cb217d8b615fd0b00c86 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cb217d8b615fd0b00c86 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cb217d8b615fd0b00c86 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:18:28,683 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cb257d8b615fd0b00c89 2025-11-30 16:18:28,685 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cb257d8b615fd0b00c89 HTTP/1.1" 502 150 2025-11-30 16:18:28,685 - lava_callback - ERROR - Error processing callback for node 6929cb257d8b615fd0b00c89: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cb257d8b615fd0b00c89 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cb257d8b615fd0b00c89 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cb257d8b615fd0b00c89 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:49686 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:18:39,471 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf4dd481a0c6f8cb5962 2025-11-30 16:18:39,472 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf4dd481a0c6f8cb5962 HTTP/1.1" 502 150 2025-11-30 16:18:39,473 - lava_callback - ERROR - Error processing callback for node 692acf4dd481a0c6f8cb5962: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf4dd481a0c6f8cb5962 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf4dd481a0c6f8cb5962 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf4dd481a0c6f8cb5962 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:34074 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:46992 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:39382 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:34754 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:34756 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:34768 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:53178 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:40418 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:35098 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:19:53,171 - lava_callback - INFO - Received callback for node 692acf4cd481a0c6f8cb595f INFO: 172.19.0.1:52748 - "POST /node/692acf4cd481a0c6f8cb595f HTTP/1.0" 202 Accepted 2025-11-30 16:19:53,178 - lava_callback - INFO - Starting async job submit for node 692acf4cd481a0c6f8cb595f 2025-11-30 16:19:53,217 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:19:53,228 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf4cd481a0c6f8cb595f HTTP/1.1" 502 150 2025-11-30 16:19:53,228 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf4cd481a0c6f8cb595f'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:19:53,228 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf4cd481a0c6f8cb595f 2025-11-30 16:19:53,228 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf4cd481a0c6f8cb595f HTTP/1.1" 502 150 2025-11-30 16:19:53,228 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf4cd481a0c6f8cb595f'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:19:55,229 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf4cd481a0c6f8cb595f 2025-11-30 16:19:55,230 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf4cd481a0c6f8cb595f HTTP/1.1" 502 150 2025-11-30 16:19:55,230 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf4cd481a0c6f8cb595f'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:52758 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:19:59,230 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf4cd481a0c6f8cb595f 2025-11-30 16:19:59,232 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf4cd481a0c6f8cb595f HTTP/1.1" 502 150 2025-11-30 16:19:59,232 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf4cd481a0c6f8cb595f'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:34172 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:20:07,232 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf4cd481a0c6f8cb595f 2025-11-30 16:20:07,234 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf4cd481a0c6f8cb595f HTTP/1.1" 502 150 2025-11-30 16:20:07,234 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf4cd481a0c6f8cb595f'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:35068 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:35070 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:35076 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:20:23,234 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf4cd481a0c6f8cb595f 2025-11-30 16:20:23,235 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf4cd481a0c6f8cb595f HTTP/1.1" 502 150 2025-11-30 16:20:23,235 - lava_callback - ERROR - Error processing callback for node 692acf4cd481a0c6f8cb595f: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf4cd481a0c6f8cb595f (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf4cd481a0c6f8cb595f (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf4cd481a0c6f8cb595f (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:40670 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:49010 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:46198 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:20:53,165 - lava_callback - INFO - Received callback for node 692acf52d481a0c6f8cb596b INFO: 172.19.0.1:49366 - "POST /node/692acf52d481a0c6f8cb596b HTTP/1.0" 202 Accepted 2025-11-30 16:20:53,171 - lava_callback - INFO - Starting async job submit for node 692acf52d481a0c6f8cb596b 2025-11-30 16:20:53,263 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:20:53,274 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf52d481a0c6f8cb596b HTTP/1.1" 502 150 2025-11-30 16:20:53,274 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf52d481a0c6f8cb596b'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:20:53,274 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf52d481a0c6f8cb596b 2025-11-30 16:20:53,274 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf52d481a0c6f8cb596b HTTP/1.1" 502 150 2025-11-30 16:20:53,274 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf52d481a0c6f8cb596b'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:20:55,275 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf52d481a0c6f8cb596b 2025-11-30 16:20:55,276 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf52d481a0c6f8cb596b HTTP/1.1" 502 150 2025-11-30 16:20:55,276 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf52d481a0c6f8cb596b'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:49368 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:20:59,276 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf52d481a0c6f8cb596b 2025-11-30 16:20:59,277 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf52d481a0c6f8cb596b HTTP/1.1" 502 150 2025-11-30 16:20:59,277 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf52d481a0c6f8cb596b'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:49240 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:21:07,278 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf52d481a0c6f8cb596b 2025-11-30 16:21:07,279 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf52d481a0c6f8cb596b HTTP/1.1" 502 150 2025-11-30 16:21:07,279 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf52d481a0c6f8cb596b'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:36988 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:36990 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:36998 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:21:23,279 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf52d481a0c6f8cb596b 2025-11-30 16:21:23,280 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf52d481a0c6f8cb596b HTTP/1.1" 502 150 2025-11-30 16:21:23,281 - lava_callback - ERROR - Error processing callback for node 692acf52d481a0c6f8cb596b: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf52d481a0c6f8cb596b (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf52d481a0c6f8cb596b (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf52d481a0c6f8cb596b (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:46248 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:48954 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:51862 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:21:53,432 - lava_callback - INFO - Received callback for node 692acf54d481a0c6f8cb596e INFO: 172.19.0.1:54208 - "POST /node/692acf54d481a0c6f8cb596e HTTP/1.0" 202 Accepted 2025-11-30 16:21:53,438 - lava_callback - INFO - Starting async job submit for node 692acf54d481a0c6f8cb596e 2025-11-30 16:21:53,463 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:21:53,477 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf54d481a0c6f8cb596e HTTP/1.1" 502 150 2025-11-30 16:21:53,477 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf54d481a0c6f8cb596e'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:21:53,477 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf54d481a0c6f8cb596e 2025-11-30 16:21:53,478 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf54d481a0c6f8cb596e HTTP/1.1" 502 150 2025-11-30 16:21:53,478 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf54d481a0c6f8cb596e'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:21:55,478 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf54d481a0c6f8cb596e 2025-11-30 16:21:55,479 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf54d481a0c6f8cb596e HTTP/1.1" 502 150 2025-11-30 16:21:55,479 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf54d481a0c6f8cb596e'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:54222 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:21:59,479 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf54d481a0c6f8cb596e 2025-11-30 16:21:59,481 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf54d481a0c6f8cb596e HTTP/1.1" 502 150 2025-11-30 16:21:59,481 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf54d481a0c6f8cb596e'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:51654 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:22:07,481 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf54d481a0c6f8cb596e 2025-11-30 16:22:07,485 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf54d481a0c6f8cb596e HTTP/1.1" 502 150 2025-11-30 16:22:07,485 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf54d481a0c6f8cb596e'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:22:09,753 - lava_callback - INFO - Received callback for node 692acf56d481a0c6f8cb5971 INFO: 172.19.0.1:50320 - "POST /node/692acf56d481a0c6f8cb5971 HTTP/1.0" 202 Accepted 2025-11-30 16:22:09,759 - lava_callback - INFO - Starting async job submit for node 692acf56d481a0c6f8cb5971 2025-11-30 16:22:09,808 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:22:09,819 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf56d481a0c6f8cb5971 HTTP/1.1" 502 150 2025-11-30 16:22:09,819 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf56d481a0c6f8cb5971'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:22:09,819 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf56d481a0c6f8cb5971 2025-11-30 16:22:09,825 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf56d481a0c6f8cb5971 HTTP/1.1" 502 150 2025-11-30 16:22:09,825 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf56d481a0c6f8cb5971'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:22:11,825 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf56d481a0c6f8cb5971 2025-11-30 16:22:11,826 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf56d481a0c6f8cb5971 HTTP/1.1" 502 150 2025-11-30 16:22:11,826 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf56d481a0c6f8cb5971'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:50334 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:50336 - "GET /api/metrics HTTP/1.0" 200 OK 2025-11-30 16:22:15,826 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf56d481a0c6f8cb5971 2025-11-30 16:22:15,828 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf56d481a0c6f8cb5971 HTTP/1.1" 502 150 2025-11-30 16:22:15,828 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf56d481a0c6f8cb5971'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:50342 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:22:23,485 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf54d481a0c6f8cb596e 2025-11-30 16:22:23,487 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf54d481a0c6f8cb596e HTTP/1.1" 502 150 2025-11-30 16:22:23,487 - lava_callback - ERROR - Error processing callback for node 692acf54d481a0c6f8cb596e: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf54d481a0c6f8cb596e (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf54d481a0c6f8cb596e (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf54d481a0c6f8cb596e (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:22:23,828 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf56d481a0c6f8cb5971 2025-11-30 16:22:23,829 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf56d481a0c6f8cb5971 HTTP/1.1" 502 150 2025-11-30 16:22:23,829 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf56d481a0c6f8cb5971'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:43280 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:22:33,639 - lava_callback - INFO - Received callback for node 692acf43d481a0c6f8cb5950 INFO: 172.19.0.1:47580 - "POST /node/692acf43d481a0c6f8cb5950 HTTP/1.0" 202 Accepted 2025-11-30 16:22:33,647 - lava_callback - INFO - Starting async job submit for node 692acf43d481a0c6f8cb5950 2025-11-30 16:22:33,783 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:22:33,793 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf43d481a0c6f8cb5950 HTTP/1.1" 502 150 2025-11-30 16:22:33,793 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf43d481a0c6f8cb5950'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:22:33,793 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf43d481a0c6f8cb5950 2025-11-30 16:22:33,794 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf43d481a0c6f8cb5950 HTTP/1.1" 502 150 2025-11-30 16:22:33,794 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf43d481a0c6f8cb5950'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:22:35,794 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf43d481a0c6f8cb5950 2025-11-30 16:22:35,795 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf43d481a0c6f8cb5950 HTTP/1.1" 502 150 2025-11-30 16:22:35,795 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf43d481a0c6f8cb5950'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:47584 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:22:39,795 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf43d481a0c6f8cb5950 2025-11-30 16:22:39,797 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf43d481a0c6f8cb5950 HTTP/1.1" 502 150 2025-11-30 16:22:39,797 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf43d481a0c6f8cb5950'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:22:39,830 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf56d481a0c6f8cb5971 2025-11-30 16:22:39,831 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf56d481a0c6f8cb5971 HTTP/1.1" 502 150 2025-11-30 16:22:39,831 - lava_callback - ERROR - Error processing callback for node 692acf56d481a0c6f8cb5971: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf56d481a0c6f8cb5971 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf56d481a0c6f8cb5971 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf56d481a0c6f8cb5971 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:60148 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:22:47,797 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf43d481a0c6f8cb5950 2025-11-30 16:22:47,798 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf43d481a0c6f8cb5950 HTTP/1.1" 502 150 2025-11-30 16:22:47,798 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692acf43d481a0c6f8cb5950'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:34540 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:23:03,798 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692acf43d481a0c6f8cb5950 2025-11-30 16:23:03,800 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692acf43d481a0c6f8cb5950 HTTP/1.1" 502 150 2025-11-30 16:23:03,800 - lava_callback - ERROR - Error processing callback for node 692acf43d481a0c6f8cb5950: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf43d481a0c6f8cb5950 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf43d481a0c6f8cb5950 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692acf43d481a0c6f8cb5950 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:45164 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:54220 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:54230 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:54244 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:59606 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:54614 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:41232 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:35932 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:44310 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:54900 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:54916 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:54928 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:53910 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:24:33,337 - lava_callback - INFO - Received callback for node 692ae8d9d481a0c6f8cb70a0 INFO: 172.19.0.1:58076 - "POST /node/692ae8d9d481a0c6f8cb70a0 HTTP/1.0" 202 Accepted 2025-11-30 16:24:33,347 - lava_callback - INFO - Starting async job submit for node 692ae8d9d481a0c6f8cb70a0 2025-11-30 16:24:33,388 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:24:33,401 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae8d9d481a0c6f8cb70a0 HTTP/1.1" 502 150 2025-11-30 16:24:33,401 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae8d9d481a0c6f8cb70a0'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:24:33,401 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae8d9d481a0c6f8cb70a0 2025-11-30 16:24:33,402 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae8d9d481a0c6f8cb70a0 HTTP/1.1" 502 150 2025-11-30 16:24:33,403 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae8d9d481a0c6f8cb70a0'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:24:35,403 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae8d9d481a0c6f8cb70a0 2025-11-30 16:24:35,404 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae8d9d481a0c6f8cb70a0 HTTP/1.1" 502 150 2025-11-30 16:24:35,404 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae8d9d481a0c6f8cb70a0'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:58090 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:24:38,248 - lava_callback - INFO - Received callback for node 6929ccbd7d8b615fd0b00ced INFO: 172.19.0.1:58094 - "POST /node/6929ccbd7d8b615fd0b00ced HTTP/1.0" 202 Accepted 2025-11-30 16:24:38,257 - lava_callback - INFO - Starting async job submit for node 6929ccbd7d8b615fd0b00ced 2025-11-30 16:24:38,258 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:24:38,273 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929ccbd7d8b615fd0b00ced HTTP/1.1" 502 150 2025-11-30 16:24:38,273 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929ccbd7d8b615fd0b00ced'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:24:38,273 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929ccbd7d8b615fd0b00ced 2025-11-30 16:24:38,279 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929ccbd7d8b615fd0b00ced HTTP/1.1" 502 150 2025-11-30 16:24:38,279 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929ccbd7d8b615fd0b00ced'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:24:38,924 - lava_callback - INFO - Received callback for node 6929ccc17d8b615fd0b00cf0 INFO: 172.19.0.1:58104 - "POST /node/6929ccc17d8b615fd0b00cf0 HTTP/1.0" 202 Accepted 2025-11-30 16:24:38,934 - lava_callback - INFO - Starting async job submit for node 6929ccc17d8b615fd0b00cf0 2025-11-30 16:24:38,935 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:24:38,953 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929ccc17d8b615fd0b00cf0 HTTP/1.1" 502 150 2025-11-30 16:24:38,953 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929ccc17d8b615fd0b00cf0'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:24:38,954 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929ccc17d8b615fd0b00cf0 2025-11-30 16:24:38,954 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929ccc17d8b615fd0b00cf0 HTTP/1.1" 502 150 2025-11-30 16:24:38,955 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929ccc17d8b615fd0b00cf0'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:24:39,404 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae8d9d481a0c6f8cb70a0 2025-11-30 16:24:39,406 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae8d9d481a0c6f8cb70a0 HTTP/1.1" 502 150 2025-11-30 16:24:39,406 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae8d9d481a0c6f8cb70a0'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:24:40,280 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929ccbd7d8b615fd0b00ced 2025-11-30 16:24:40,281 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929ccbd7d8b615fd0b00ced HTTP/1.1" 502 150 2025-11-30 16:24:40,281 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929ccbd7d8b615fd0b00ced'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:24:40,955 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929ccc17d8b615fd0b00cf0 2025-11-30 16:24:40,956 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929ccc17d8b615fd0b00cf0 HTTP/1.1" 502 150 2025-11-30 16:24:40,956 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929ccc17d8b615fd0b00cf0'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:24:44,281 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929ccbd7d8b615fd0b00ced 2025-11-30 16:24:44,282 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929ccbd7d8b615fd0b00ced HTTP/1.1" 502 150 2025-11-30 16:24:44,283 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929ccbd7d8b615fd0b00ced'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:24:44,956 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929ccc17d8b615fd0b00cf0 2025-11-30 16:24:44,957 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929ccc17d8b615fd0b00cf0 HTTP/1.1" 502 150 2025-11-30 16:24:44,957 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929ccc17d8b615fd0b00cf0'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:57772 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:24:47,406 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae8d9d481a0c6f8cb70a0 2025-11-30 16:24:47,407 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae8d9d481a0c6f8cb70a0 HTTP/1.1" 502 150 2025-11-30 16:24:47,407 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae8d9d481a0c6f8cb70a0'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:24:52,283 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929ccbd7d8b615fd0b00ced 2025-11-30 16:24:52,284 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929ccbd7d8b615fd0b00ced HTTP/1.1" 502 150 2025-11-30 16:24:52,284 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929ccbd7d8b615fd0b00ced'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:24:52,958 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929ccc17d8b615fd0b00cf0 2025-11-30 16:24:52,959 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929ccc17d8b615fd0b00cf0 HTTP/1.1" 502 150 2025-11-30 16:24:52,959 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929ccc17d8b615fd0b00cf0'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:24:53,531 - lava_callback - INFO - Received callback for node 692ae425d481a0c6f8cb6ecb INFO: 172.19.0.1:59294 - "POST /node/692ae425d481a0c6f8cb6ecb HTTP/1.0" 202 Accepted 2025-11-30 16:24:53,545 - lava_callback - INFO - Starting async job submit for node 692ae425d481a0c6f8cb6ecb 2025-11-30 16:24:53,617 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:24:53,633 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae425d481a0c6f8cb6ecb HTTP/1.1" 502 150 2025-11-30 16:24:53,634 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae425d481a0c6f8cb6ecb'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:24:53,634 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae425d481a0c6f8cb6ecb 2025-11-30 16:24:53,634 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae425d481a0c6f8cb6ecb HTTP/1.1" 502 150 2025-11-30 16:24:53,635 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae425d481a0c6f8cb6ecb'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:24:55,635 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae425d481a0c6f8cb6ecb 2025-11-30 16:24:55,636 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae425d481a0c6f8cb6ecb HTTP/1.1" 502 150 2025-11-30 16:24:55,636 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae425d481a0c6f8cb6ecb'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:59306 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:24:59,636 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae425d481a0c6f8cb6ecb 2025-11-30 16:24:59,637 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae425d481a0c6f8cb6ecb HTTP/1.1" 502 150 2025-11-30 16:24:59,638 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae425d481a0c6f8cb6ecb'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:25:03,408 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae8d9d481a0c6f8cb70a0 2025-11-30 16:25:03,409 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae8d9d481a0c6f8cb70a0 HTTP/1.1" 502 150 2025-11-30 16:25:03,409 - lava_callback - ERROR - Error processing callback for node 692ae8d9d481a0c6f8cb70a0: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae8d9d481a0c6f8cb70a0 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae8d9d481a0c6f8cb70a0 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae8d9d481a0c6f8cb70a0 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:51198 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:25:07,638 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae425d481a0c6f8cb6ecb 2025-11-30 16:25:07,639 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae425d481a0c6f8cb6ecb HTTP/1.1" 502 150 2025-11-30 16:25:07,639 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae425d481a0c6f8cb6ecb'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:25:08,285 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929ccbd7d8b615fd0b00ced 2025-11-30 16:25:08,287 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929ccbd7d8b615fd0b00ced HTTP/1.1" 502 150 2025-11-30 16:25:08,287 - lava_callback - ERROR - Error processing callback for node 6929ccbd7d8b615fd0b00ced: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929ccbd7d8b615fd0b00ced (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929ccbd7d8b615fd0b00ced (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929ccbd7d8b615fd0b00ced (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:25:08,959 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929ccc17d8b615fd0b00cf0 2025-11-30 16:25:08,960 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929ccc17d8b615fd0b00cf0 HTTP/1.1" 502 150 2025-11-30 16:25:08,961 - lava_callback - ERROR - Error processing callback for node 6929ccc17d8b615fd0b00cf0: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929ccc17d8b615fd0b00cf0 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929ccc17d8b615fd0b00cf0 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929ccc17d8b615fd0b00cf0 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:42074 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:42084 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:42096 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:25:23,639 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae425d481a0c6f8cb6ecb 2025-11-30 16:25:23,640 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae425d481a0c6f8cb6ecb HTTP/1.1" 502 150 2025-11-30 16:25:23,641 - lava_callback - ERROR - Error processing callback for node 692ae425d481a0c6f8cb6ecb: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae425d481a0c6f8cb6ecb (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae425d481a0c6f8cb6ecb (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae425d481a0c6f8cb6ecb (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:50706 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:44242 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:47684 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:57300 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:25:56,611 - lava_callback - INFO - Received callback for node 692a09107d8b615fd0b029b5 INFO: 172.19.0.1:57306 - "POST /node/692a09107d8b615fd0b029b5 HTTP/1.0" 202 Accepted 2025-11-30 16:25:56,617 - lava_callback - INFO - Starting async job submit for node 692a09107d8b615fd0b029b5 2025-11-30 16:25:56,654 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:25:56,690 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a09107d8b615fd0b029b5 HTTP/1.1" 502 150 2025-11-30 16:25:56,690 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a09107d8b615fd0b029b5'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:25:56,690 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a09107d8b615fd0b029b5 2025-11-30 16:25:56,691 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a09107d8b615fd0b029b5 HTTP/1.1" 502 150 2025-11-30 16:25:56,691 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a09107d8b615fd0b029b5'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:25:58,691 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a09107d8b615fd0b029b5 2025-11-30 16:25:58,692 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a09107d8b615fd0b029b5 HTTP/1.1" 502 150 2025-11-30 16:25:58,692 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a09107d8b615fd0b029b5'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:26:02,693 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a09107d8b615fd0b029b5 2025-11-30 16:26:02,694 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a09107d8b615fd0b029b5 HTTP/1.1" 502 150 2025-11-30 16:26:02,694 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a09107d8b615fd0b029b5'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:35206 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:26:10,694 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a09107d8b615fd0b029b5 2025-11-30 16:26:10,695 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a09107d8b615fd0b029b5 HTTP/1.1" 502 150 2025-11-30 16:26:10,695 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a09107d8b615fd0b029b5'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:60890 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:26:13,385 - lava_callback - INFO - Received callback for node 692ae59ad481a0c6f8cb6f3c INFO: 172.19.0.1:60894 - "POST /node/692ae59ad481a0c6f8cb6f3c HTTP/1.0" 202 Accepted 2025-11-30 16:26:13,397 - lava_callback - INFO - Starting async job submit for node 692ae59ad481a0c6f8cb6f3c 2025-11-30 16:26:13,504 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:26:13,524 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae59ad481a0c6f8cb6f3c HTTP/1.1" 502 150 2025-11-30 16:26:13,524 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae59ad481a0c6f8cb6f3c'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:26:13,524 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae59ad481a0c6f8cb6f3c 2025-11-30 16:26:13,525 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae59ad481a0c6f8cb6f3c HTTP/1.1" 502 150 2025-11-30 16:26:13,525 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae59ad481a0c6f8cb6f3c'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:60900 - "GET /api/metrics HTTP/1.0" 200 OK 2025-11-30 16:26:15,525 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae59ad481a0c6f8cb6f3c 2025-11-30 16:26:15,527 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae59ad481a0c6f8cb6f3c HTTP/1.1" 502 150 2025-11-30 16:26:15,527 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae59ad481a0c6f8cb6f3c'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:60912 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:26:19,527 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae59ad481a0c6f8cb6f3c 2025-11-30 16:26:19,534 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae59ad481a0c6f8cb6f3c HTTP/1.1" 502 150 2025-11-30 16:26:19,535 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae59ad481a0c6f8cb6f3c'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:60988 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:26:26,696 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a09107d8b615fd0b029b5 2025-11-30 16:26:26,697 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a09107d8b615fd0b029b5 HTTP/1.1" 502 150 2025-11-30 16:26:26,697 - lava_callback - ERROR - Error processing callback for node 692a09107d8b615fd0b029b5: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692a09107d8b615fd0b029b5 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692a09107d8b615fd0b029b5 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692a09107d8b615fd0b029b5 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:26:27,535 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae59ad481a0c6f8cb6f3c 2025-11-30 16:26:27,536 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae59ad481a0c6f8cb6f3c HTTP/1.1" 502 150 2025-11-30 16:26:27,536 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae59ad481a0c6f8cb6f3c'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:26:33,791 - lava_callback - INFO - Received callback for node 692ad957d481a0c6f8cb6370 INFO: 172.19.0.1:38826 - "POST /node/692ad957d481a0c6f8cb6370 HTTP/1.0" 202 Accepted 2025-11-30 16:26:33,798 - lava_callback - INFO - Starting async job submit for node 692ad957d481a0c6f8cb6370 2025-11-30 16:26:33,831 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:26:33,840 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ad957d481a0c6f8cb6370 HTTP/1.1" 502 150 2025-11-30 16:26:33,840 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ad957d481a0c6f8cb6370'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:26:33,840 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ad957d481a0c6f8cb6370 2025-11-30 16:26:33,841 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ad957d481a0c6f8cb6370 HTTP/1.1" 502 150 2025-11-30 16:26:33,841 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ad957d481a0c6f8cb6370'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:26:35,841 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ad957d481a0c6f8cb6370 2025-11-30 16:26:35,842 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ad957d481a0c6f8cb6370 HTTP/1.1" 502 150 2025-11-30 16:26:35,843 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ad957d481a0c6f8cb6370'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:38830 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:26:39,843 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ad957d481a0c6f8cb6370 2025-11-30 16:26:39,844 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ad957d481a0c6f8cb6370 HTTP/1.1" 502 150 2025-11-30 16:26:39,844 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ad957d481a0c6f8cb6370'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:26:43,536 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae59ad481a0c6f8cb6f3c 2025-11-30 16:26:43,537 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae59ad481a0c6f8cb6f3c HTTP/1.1" 502 150 2025-11-30 16:26:43,538 - lava_callback - ERROR - Error processing callback for node 692ae59ad481a0c6f8cb6f3c: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae59ad481a0c6f8cb6f3c (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae59ad481a0c6f8cb6f3c (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae59ad481a0c6f8cb6f3c (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:45924 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:26:47,844 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ad957d481a0c6f8cb6370 2025-11-30 16:26:47,845 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ad957d481a0c6f8cb6370 HTTP/1.1" 502 150 2025-11-30 16:26:47,846 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ad957d481a0c6f8cb6370'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:39456 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:27:03,846 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ad957d481a0c6f8cb6370 2025-11-30 16:27:03,847 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ad957d481a0c6f8cb6370 HTTP/1.1" 502 150 2025-11-30 16:27:03,847 - lava_callback - ERROR - Error processing callback for node 692ad957d481a0c6f8cb6370: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ad957d481a0c6f8cb6370 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ad957d481a0c6f8cb6370 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ad957d481a0c6f8cb6370 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:44970 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:27:08,226 - lava_callback - INFO - Received callback for node 6929cd4c7d8b615fd0b00daf INFO: 172.19.0.1:44976 - "POST /node/6929cd4c7d8b615fd0b00daf HTTP/1.0" 202 Accepted 2025-11-30 16:27:08,231 - lava_callback - INFO - Starting async job submit for node 6929cd4c7d8b615fd0b00daf 2025-11-30 16:27:08,232 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:27:08,253 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cd4c7d8b615fd0b00daf HTTP/1.1" 502 150 2025-11-30 16:27:08,253 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cd4c7d8b615fd0b00daf'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:27:08,253 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cd4c7d8b615fd0b00daf 2025-11-30 16:27:08,254 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cd4c7d8b615fd0b00daf HTTP/1.1" 502 150 2025-11-30 16:27:08,254 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cd4c7d8b615fd0b00daf'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:27:10,254 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cd4c7d8b615fd0b00daf 2025-11-30 16:27:10,255 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cd4c7d8b615fd0b00daf HTTP/1.1" 502 150 2025-11-30 16:27:10,255 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cd4c7d8b615fd0b00daf'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:53328 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:27:14,256 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cd4c7d8b615fd0b00daf 2025-11-30 16:27:14,257 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cd4c7d8b615fd0b00daf HTTP/1.1" 502 150 2025-11-30 16:27:14,257 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cd4c7d8b615fd0b00daf'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:53340 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:53346 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:27:22,257 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cd4c7d8b615fd0b00daf 2025-11-30 16:27:22,261 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cd4c7d8b615fd0b00daf HTTP/1.1" 502 150 2025-11-30 16:27:22,261 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6929cd4c7d8b615fd0b00daf'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:50024 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:43408 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:27:38,261 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6929cd4c7d8b615fd0b00daf 2025-11-30 16:27:38,262 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6929cd4c7d8b615fd0b00daf HTTP/1.1" 502 150 2025-11-30 16:27:38,263 - lava_callback - ERROR - Error processing callback for node 6929cd4c7d8b615fd0b00daf: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cd4c7d8b615fd0b00daf (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cd4c7d8b615fd0b00daf (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6929cd4c7d8b615fd0b00daf (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:57308 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:41162 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:50960 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:42428 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:42430 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:42434 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:42790 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:48528 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:40106 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:28:49,783 - lava_callback - INFO - Received callback for node 692ae6bad481a0c6f8cb6fb7 INFO: 172.19.0.1:38452 - "POST /node/692ae6bad481a0c6f8cb6fb7 HTTP/1.0" 202 Accepted 2025-11-30 16:28:49,790 - lava_callback - INFO - Starting async job submit for node 692ae6bad481a0c6f8cb6fb7 2025-11-30 16:28:49,831 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:28:49,860 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae6bad481a0c6f8cb6fb7 HTTP/1.1" 502 150 2025-11-30 16:28:49,861 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae6bad481a0c6f8cb6fb7'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:28:49,861 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae6bad481a0c6f8cb6fb7 2025-11-30 16:28:49,862 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae6bad481a0c6f8cb6fb7 HTTP/1.1" 502 150 2025-11-30 16:28:49,862 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae6bad481a0c6f8cb6fb7'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:28:51,862 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae6bad481a0c6f8cb6fb7 2025-11-30 16:28:51,863 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae6bad481a0c6f8cb6fb7 HTTP/1.1" 502 150 2025-11-30 16:28:51,863 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae6bad481a0c6f8cb6fb7'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:28:55,864 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae6bad481a0c6f8cb6fb7 2025-11-30 16:28:55,865 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae6bad481a0c6f8cb6fb7 HTTP/1.1" 502 150 2025-11-30 16:28:55,865 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae6bad481a0c6f8cb6fb7'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:38466 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:29:03,865 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae6bad481a0c6f8cb6fb7 2025-11-30 16:29:03,866 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae6bad481a0c6f8cb6fb7 HTTP/1.1" 502 150 2025-11-30 16:29:03,867 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae6bad481a0c6f8cb6fb7'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:35390 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:45690 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:45704 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:45710 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:29:19,867 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae6bad481a0c6f8cb6fb7 2025-11-30 16:29:19,868 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae6bad481a0c6f8cb6fb7 HTTP/1.1" 502 150 2025-11-30 16:29:19,868 - lava_callback - ERROR - Error processing callback for node 692ae6bad481a0c6f8cb6fb7: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae6bad481a0c6f8cb6fb7 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae6bad481a0c6f8cb6fb7 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae6bad481a0c6f8cb6fb7 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:39600 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:29:33,637 - lava_callback - INFO - Received callback for node 692ae91fd481a0c6f8cb70d1 INFO: 172.19.0.1:46340 - "POST /node/692ae91fd481a0c6f8cb70d1 HTTP/1.0" 202 Accepted 2025-11-30 16:29:33,651 - lava_callback - INFO - Starting async job submit for node 692ae91fd481a0c6f8cb70d1 2025-11-30 16:29:33,905 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:29:33,924 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae91fd481a0c6f8cb70d1 HTTP/1.1" 502 150 2025-11-30 16:29:33,924 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae91fd481a0c6f8cb70d1'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:29:33,924 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae91fd481a0c6f8cb70d1 2025-11-30 16:29:33,925 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae91fd481a0c6f8cb70d1 HTTP/1.1" 502 150 2025-11-30 16:29:33,925 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae91fd481a0c6f8cb70d1'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:29:35,925 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae91fd481a0c6f8cb70d1 2025-11-30 16:29:35,928 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae91fd481a0c6f8cb70d1 HTTP/1.1" 502 150 2025-11-30 16:29:35,929 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae91fd481a0c6f8cb70d1'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:46356 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:29:39,929 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae91fd481a0c6f8cb70d1 2025-11-30 16:29:39,930 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae91fd481a0c6f8cb70d1 HTTP/1.1" 502 150 2025-11-30 16:29:39,930 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae91fd481a0c6f8cb70d1'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:40694 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:29:47,930 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae91fd481a0c6f8cb70d1 2025-11-30 16:29:47,931 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae91fd481a0c6f8cb70d1 HTTP/1.1" 502 150 2025-11-30 16:29:47,931 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae91fd481a0c6f8cb70d1'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:29:53,712 - lava_callback - INFO - Received callback for node 692ae91dd481a0c6f8cb70ce INFO: 172.19.0.1:56102 - "POST /node/692ae91dd481a0c6f8cb70ce HTTP/1.0" 202 Accepted 2025-11-30 16:29:53,719 - lava_callback - INFO - Starting async job submit for node 692ae91dd481a0c6f8cb70ce 2025-11-30 16:29:53,744 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:29:53,754 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae91dd481a0c6f8cb70ce HTTP/1.1" 502 150 2025-11-30 16:29:53,754 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae91dd481a0c6f8cb70ce'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:29:53,754 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae91dd481a0c6f8cb70ce 2025-11-30 16:29:53,755 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae91dd481a0c6f8cb70ce HTTP/1.1" 502 150 2025-11-30 16:29:53,755 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae91dd481a0c6f8cb70ce'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:29:55,755 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae91dd481a0c6f8cb70ce 2025-11-30 16:29:55,756 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae91dd481a0c6f8cb70ce HTTP/1.1" 502 150 2025-11-30 16:29:55,756 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae91dd481a0c6f8cb70ce'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:56118 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:29:59,756 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae91dd481a0c6f8cb70ce 2025-11-30 16:29:59,758 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae91dd481a0c6f8cb70ce HTTP/1.1" 502 150 2025-11-30 16:29:59,759 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae91dd481a0c6f8cb70ce'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:30:03,932 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae91fd481a0c6f8cb70d1 2025-11-30 16:30:03,933 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae91fd481a0c6f8cb70d1 HTTP/1.1" 502 150 2025-11-30 16:30:03,933 - lava_callback - ERROR - Error processing callback for node 692ae91fd481a0c6f8cb70d1: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae91fd481a0c6f8cb70d1 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae91fd481a0c6f8cb70d1 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae91fd481a0c6f8cb70d1 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:53228 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:30:07,759 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae91dd481a0c6f8cb70ce 2025-11-30 16:30:07,760 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae91dd481a0c6f8cb70ce HTTP/1.1" 502 150 2025-11-30 16:30:07,760 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae91dd481a0c6f8cb70ce'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:42594 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:42606 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:42614 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:30:23,760 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae91dd481a0c6f8cb70ce 2025-11-30 16:30:23,762 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae91dd481a0c6f8cb70ce HTTP/1.1" 502 150 2025-11-30 16:30:23,762 - lava_callback - ERROR - Error processing callback for node 692ae91dd481a0c6f8cb70ce: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae91dd481a0c6f8cb70ce (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae91dd481a0c6f8cb70ce (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae91dd481a0c6f8cb70ce (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:55596 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:60456 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:30:37,786 - lava_callback - INFO - Received callback for node 692a767af8f845693da45a68 INFO: 172.19.0.1:60470 - "POST /node/692a767af8f845693da45a68 HTTP/1.0" 202 Accepted 2025-11-30 16:30:37,797 - lava_callback - INFO - Starting async job submit for node 692a767af8f845693da45a68 2025-11-30 16:30:37,826 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:30:37,839 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a767af8f845693da45a68 HTTP/1.1" 502 150 2025-11-30 16:30:37,839 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a767af8f845693da45a68'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:30:37,839 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a767af8f845693da45a68 2025-11-30 16:30:37,840 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a767af8f845693da45a68 HTTP/1.1" 502 150 2025-11-30 16:30:37,840 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a767af8f845693da45a68'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:30:39,840 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a767af8f845693da45a68 2025-11-30 16:30:39,842 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a767af8f845693da45a68 HTTP/1.1" 502 150 2025-11-30 16:30:39,842 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a767af8f845693da45a68'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:30:43,842 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a767af8f845693da45a68 2025-11-30 16:30:43,843 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a767af8f845693da45a68 HTTP/1.1" 502 150 2025-11-30 16:30:43,843 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a767af8f845693da45a68'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:55294 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:30:51,844 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a767af8f845693da45a68 2025-11-30 16:30:51,845 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a767af8f845693da45a68 HTTP/1.1" 502 150 2025-11-30 16:30:51,845 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a767af8f845693da45a68'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:48108 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:58590 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:31:07,845 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a767af8f845693da45a68 2025-11-30 16:31:07,847 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a767af8f845693da45a68 HTTP/1.1" 502 150 2025-11-30 16:31:07,847 - lava_callback - ERROR - Error processing callback for node 692a767af8f845693da45a68: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692a767af8f845693da45a68 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692a767af8f845693da45a68 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692a767af8f845693da45a68 (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:31:11,660 - lava_callback - INFO - Received callback for node 692a6371f8f845693da45380 INFO: 172.19.0.1:44692 - "POST /node/692a6371f8f845693da45380 HTTP/1.0" 202 Accepted 2025-11-30 16:31:11,669 - lava_callback - INFO - Starting async job submit for node 692a6371f8f845693da45380 2025-11-30 16:31:11,791 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:31:11,810 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a6371f8f845693da45380 HTTP/1.1" 502 150 2025-11-30 16:31:11,810 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a6371f8f845693da45380'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:31:11,810 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a6371f8f845693da45380 2025-11-30 16:31:11,811 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a6371f8f845693da45380 HTTP/1.1" 502 150 2025-11-30 16:31:11,811 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a6371f8f845693da45380'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:44702 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:31:13,811 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a6371f8f845693da45380 2025-11-30 16:31:13,813 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a6371f8f845693da45380 HTTP/1.1" 502 150 2025-11-30 16:31:13,813 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a6371f8f845693da45380'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:44710 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:44722 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:31:17,813 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a6371f8f845693da45380 2025-11-30 16:31:17,814 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a6371f8f845693da45380 HTTP/1.1" 502 150 2025-11-30 16:31:17,814 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a6371f8f845693da45380'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:31:25,815 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a6371f8f845693da45380 2025-11-30 16:31:25,816 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a6371f8f845693da45380 HTTP/1.1" 502 150 2025-11-30 16:31:25,816 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a6371f8f845693da45380'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:42652 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:50968 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:31:41,816 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a6371f8f845693da45380 2025-11-30 16:31:41,817 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a6371f8f845693da45380 HTTP/1.1" 502 150 2025-11-30 16:31:41,818 - lava_callback - ERROR - Error processing callback for node 692a6371f8f845693da45380: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692a6371f8f845693da45380 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692a6371f8f845693da45380 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692a6371f8f845693da45380 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:48798 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:41164 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:42504 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:56012 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:56020 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:56028 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:37644 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:36738 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:50618 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:53520 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:57832 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:38954 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:38962 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:38966 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:34596 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:33:33,569 - lava_callback - INFO - Received callback for node 692ae926d481a0c6f8cb70dd INFO: 172.19.0.1:59874 - "POST /node/692ae926d481a0c6f8cb70dd HTTP/1.0" 202 Accepted 2025-11-30 16:33:33,579 - lava_callback - INFO - Starting async job submit for node 692ae926d481a0c6f8cb70dd 2025-11-30 16:33:33,626 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:33:33,642 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae926d481a0c6f8cb70dd HTTP/1.1" 502 150 2025-11-30 16:33:33,642 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae926d481a0c6f8cb70dd'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:33:33,643 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae926d481a0c6f8cb70dd 2025-11-30 16:33:33,643 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae926d481a0c6f8cb70dd HTTP/1.1" 502 150 2025-11-30 16:33:33,643 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae926d481a0c6f8cb70dd'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:33:35,644 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae926d481a0c6f8cb70dd 2025-11-30 16:33:35,645 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae926d481a0c6f8cb70dd HTTP/1.1" 502 150 2025-11-30 16:33:35,645 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae926d481a0c6f8cb70dd'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:59880 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:33:39,645 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae926d481a0c6f8cb70dd 2025-11-30 16:33:39,646 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae926d481a0c6f8cb70dd HTTP/1.1" 502 150 2025-11-30 16:33:39,646 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae926d481a0c6f8cb70dd'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:49850 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:33:47,646 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae926d481a0c6f8cb70dd 2025-11-30 16:33:47,647 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae926d481a0c6f8cb70dd HTTP/1.1" 502 150 2025-11-30 16:33:47,648 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae926d481a0c6f8cb70dd'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:33024 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:34:03,648 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae926d481a0c6f8cb70dd 2025-11-30 16:34:03,649 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae926d481a0c6f8cb70dd HTTP/1.1" 502 150 2025-11-30 16:34:03,649 - lava_callback - ERROR - Error processing callback for node 692ae926d481a0c6f8cb70dd: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae926d481a0c6f8cb70dd (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae926d481a0c6f8cb70dd (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae926d481a0c6f8cb70dd (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:37042 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:46786 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:46792 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:46794 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:51686 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:60258 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:57024 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:40658 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:37234 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:35652 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:35658 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:35674 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:36126 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:35:30,059 - lava_callback - INFO - Received callback for node 692ae924d481a0c6f8cb70da INFO: 172.19.0.1:42704 - "POST /node/692ae924d481a0c6f8cb70da HTTP/1.0" 202 Accepted 2025-11-30 16:35:30,066 - lava_callback - INFO - Starting async job submit for node 692ae924d481a0c6f8cb70da 2025-11-30 16:35:30,197 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:35:30,217 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae924d481a0c6f8cb70da HTTP/1.1" 502 150 2025-11-30 16:35:30,217 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae924d481a0c6f8cb70da'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:35:30,217 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae924d481a0c6f8cb70da 2025-11-30 16:35:30,218 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae924d481a0c6f8cb70da HTTP/1.1" 502 150 2025-11-30 16:35:30,218 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae924d481a0c6f8cb70da'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:35:32,219 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae924d481a0c6f8cb70da 2025-11-30 16:35:32,220 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae924d481a0c6f8cb70da HTTP/1.1" 502 150 2025-11-30 16:35:32,220 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae924d481a0c6f8cb70da'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:35:34,227 - lava_callback - INFO - Received callback for node 692ae921d481a0c6f8cb70d4 INFO: 172.19.0.1:42718 - "POST /node/692ae921d481a0c6f8cb70d4 HTTP/1.0" 202 Accepted 2025-11-30 16:35:34,243 - lava_callback - INFO - Starting async job submit for node 692ae921d481a0c6f8cb70d4 2025-11-30 16:35:34,783 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:35:34,799 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae921d481a0c6f8cb70d4 HTTP/1.1" 502 150 2025-11-30 16:35:34,799 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae921d481a0c6f8cb70d4'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:35:34,799 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae921d481a0c6f8cb70d4 2025-11-30 16:35:34,800 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae921d481a0c6f8cb70d4 HTTP/1.1" 502 150 2025-11-30 16:35:34,800 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae921d481a0c6f8cb70d4'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:35:36,220 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae924d481a0c6f8cb70da 2025-11-30 16:35:36,221 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae924d481a0c6f8cb70da HTTP/1.1" 502 150 2025-11-30 16:35:36,221 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae924d481a0c6f8cb70da'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:42722 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:35:36,801 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae921d481a0c6f8cb70d4 2025-11-30 16:35:36,802 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae921d481a0c6f8cb70d4 HTTP/1.1" 502 150 2025-11-30 16:35:36,802 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae921d481a0c6f8cb70d4'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:35:40,802 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae921d481a0c6f8cb70d4 2025-11-30 16:35:40,803 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae921d481a0c6f8cb70d4 HTTP/1.1" 502 150 2025-11-30 16:35:40,804 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae921d481a0c6f8cb70d4'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:35:44,222 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae924d481a0c6f8cb70da 2025-11-30 16:35:44,223 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae924d481a0c6f8cb70da HTTP/1.1" 502 150 2025-11-30 16:35:44,223 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae924d481a0c6f8cb70da'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:34516 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:35:48,804 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae921d481a0c6f8cb70d4 2025-11-30 16:35:48,805 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae921d481a0c6f8cb70d4 HTTP/1.1" 502 150 2025-11-30 16:35:48,805 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae921d481a0c6f8cb70d4'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:55216 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:36:00,223 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae924d481a0c6f8cb70da 2025-11-30 16:36:00,224 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae924d481a0c6f8cb70da HTTP/1.1" 502 150 2025-11-30 16:36:00,225 - lava_callback - ERROR - Error processing callback for node 692ae924d481a0c6f8cb70da: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae924d481a0c6f8cb70da (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae924d481a0c6f8cb70da (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae924d481a0c6f8cb70da (Caused by ResponseError('too many 502 error responses')) 2025-11-30 16:36:04,805 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae921d481a0c6f8cb70d4 2025-11-30 16:36:04,807 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae921d481a0c6f8cb70d4 HTTP/1.1" 502 150 2025-11-30 16:36:04,807 - lava_callback - ERROR - Error processing callback for node 692ae921d481a0c6f8cb70d4: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae921d481a0c6f8cb70d4 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae921d481a0c6f8cb70d4 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae921d481a0c6f8cb70d4 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:40118 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:48920 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:48932 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:48948 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:41262 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:47936 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:47192 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:58004 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:54700 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:60118 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:60120 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:60132 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:48334 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:37:33,766 - lava_callback - INFO - Received callback for node 692ae929d481a0c6f8cb70e3 INFO: 172.19.0.1:49900 - "POST /node/692ae929d481a0c6f8cb70e3 HTTP/1.0" 202 Accepted 2025-11-30 16:37:33,772 - lava_callback - INFO - Starting async job submit for node 692ae929d481a0c6f8cb70e3 2025-11-30 16:37:33,797 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:37:33,809 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae929d481a0c6f8cb70e3 HTTP/1.1" 502 150 2025-11-30 16:37:33,809 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae929d481a0c6f8cb70e3'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:37:33,809 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae929d481a0c6f8cb70e3 2025-11-30 16:37:33,810 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae929d481a0c6f8cb70e3 HTTP/1.1" 502 150 2025-11-30 16:37:33,810 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae929d481a0c6f8cb70e3'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:37:35,810 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae929d481a0c6f8cb70e3 2025-11-30 16:37:35,811 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae929d481a0c6f8cb70e3 HTTP/1.1" 502 150 2025-11-30 16:37:35,811 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae929d481a0c6f8cb70e3'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:49904 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:37:39,811 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae929d481a0c6f8cb70e3 2025-11-30 16:37:39,812 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae929d481a0c6f8cb70e3 HTTP/1.1" 502 150 2025-11-30 16:37:39,813 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae929d481a0c6f8cb70e3'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:59818 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:37:47,813 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae929d481a0c6f8cb70e3 2025-11-30 16:37:47,820 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae929d481a0c6f8cb70e3 HTTP/1.1" 502 150 2025-11-30 16:37:47,820 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692ae929d481a0c6f8cb70e3'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:45262 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:38:03,820 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692ae929d481a0c6f8cb70e3 2025-11-30 16:38:03,822 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692ae929d481a0c6f8cb70e3 HTTP/1.1" 502 150 2025-11-30 16:38:03,822 - lava_callback - ERROR - Error processing callback for node 692ae929d481a0c6f8cb70e3: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae929d481a0c6f8cb70e3 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae929d481a0c6f8cb70e3 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/692ae929d481a0c6f8cb70e3 (Caused by ResponseError('too many 502 error responses')) INFO: 172.19.0.1:51802 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:41314 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:41318 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.19.0.1:41324 - "GET / HTTP/1.0" 200 OK INFO: 172.19.0.1:40954 - "GET / HTTP/1.0" 200 OK 2025-11-30 16:38:33,524 - lava_callback - INFO - Received callback for node 692a7495f8f845693da4583a INFO: 172.19.0.1:58692 - "POST /node/692a7495f8f845693da4583a HTTP/1.0" 202 Accepted 2025-11-30 16:38:33,534 - lava_callback - INFO - Starting async job submit for node 692a7495f8f845693da4583a 2025-11-30 16:38:33,587 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-30 16:38:33,600 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a7495f8f845693da4583a HTTP/1.1" 502 150 2025-11-30 16:38:33,600 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a7495f8f845693da4583a'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:38:33,600 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a7495f8f845693da4583a 2025-11-30 16:38:33,601 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a7495f8f845693da4583a HTTP/1.1" 502 150 2025-11-30 16:38:33,601 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a7495f8f845693da4583a'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:38:35,601 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a7495f8f845693da4583a 2025-11-30 16:38:35,730 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a7495f8f845693da4583a HTTP/1.1" 502 150 2025-11-30 16:38:35,730 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a7495f8f845693da4583a'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.19.0.1:58706 - "GET / HTTP/1.0" 200 OK INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [1] 2025-11-30 16:38:39,730 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a7495f8f845693da4583a 2025-11-30 16:38:39,732 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a7495f8f845693da4583a HTTP/1.1" 502 150 2025-11-30 16:38:39,732 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a7495f8f845693da4583a'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-11-30 16:38:47,732 - urllib3.connectionpool - DEBUG - Retry: /latest/node/692a7495f8f845693da4583a 2025-11-30 16:38:47,734 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/692a7495f8f845693da4583a HTTP/1.1" 502 150 2025-11-30 16:38:47,734 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/692a7495f8f845693da4583a'): Retry(total=0, connect=None, read=None, redirect=None, status=None)