INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) INFO: 172.21.0.1:59976 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57920 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:57924 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:36:17,304 - lava_callback - INFO - Received callback for node 693b12c9fee8383e92b73e6d 2025-12-11 19:36:17,325 - lava_callback - INFO - Starting async job submit for node 693b12c9fee8383e92b73e6d INFO: 172.21.0.1:57928 - "POST /node/693b12c9fee8383e92b73e6d HTTP/1.0" 202 Accepted 2025-12-11 19:36:17,364 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:36:17,383 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b12c9fee8383e92b73e6d HTTP/1.1" 502 150 2025-12-11 19:36:17,384 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b12c9fee8383e92b73e6d'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:36:17,384 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b12c9fee8383e92b73e6d 2025-12-11 19:36:17,384 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b12c9fee8383e92b73e6d HTTP/1.1" 502 150 2025-12-11 19:36:17,385 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b12c9fee8383e92b73e6d'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:36:19,385 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b12c9fee8383e92b73e6d 2025-12-11 19:36:19,389 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b12c9fee8383e92b73e6d HTTP/1.1" 502 150 2025-12-11 19:36:19,389 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b12c9fee8383e92b73e6d'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:36:23,389 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b12c9fee8383e92b73e6d 2025-12-11 19:36:23,394 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b12c9fee8383e92b73e6d HTTP/1.1" 502 150 2025-12-11 19:36:23,394 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b12c9fee8383e92b73e6d'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:47294 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47300 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:36:31,394 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b12c9fee8383e92b73e6d 2025-12-11 19:36:31,399 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b12c9fee8383e92b73e6d HTTP/1.1" 502 150 2025-12-11 19:36:31,400 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b12c9fee8383e92b73e6d'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:36:35,292 - lava_callback - INFO - Received callback for node 693b17f315f407b7629df39a 2025-12-11 19:36:35,305 - lava_callback - INFO - Starting async job submit for node 693b17f315f407b7629df39a INFO: 172.21.0.1:46758 - "POST /node/693b17f315f407b7629df39a HTTP/1.0" 202 Accepted 2025-12-11 19:36:35,394 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:36:35,428 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b17f315f407b7629df39a HTTP/1.1" 502 150 2025-12-11 19:36:35,428 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b17f315f407b7629df39a'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:36:35,428 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b17f315f407b7629df39a 2025-12-11 19:36:35,431 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b17f315f407b7629df39a HTTP/1.1" 502 150 2025-12-11 19:36:35,432 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b17f315f407b7629df39a'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:46774 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:36:37,432 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b17f315f407b7629df39a 2025-12-11 19:36:37,433 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b17f315f407b7629df39a HTTP/1.1" 502 150 2025-12-11 19:36:37,433 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b17f315f407b7629df39a'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:36:40,997 - lava_callback - INFO - Received callback for node 693b03a6fee8383e92b6f516 2025-12-11 19:36:41,013 - lava_callback - INFO - Starting async job submit for node 693b03a6fee8383e92b6f516 INFO: 172.21.0.1:43338 - "POST /node/693b03a6fee8383e92b6f516 HTTP/1.0" 202 Accepted 2025-12-11 19:36:41,189 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:36:41,208 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b03a6fee8383e92b6f516 HTTP/1.1" 502 150 2025-12-11 19:36:41,208 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b03a6fee8383e92b6f516'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:36:41,208 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b03a6fee8383e92b6f516 2025-12-11 19:36:41,209 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b03a6fee8383e92b6f516 HTTP/1.1" 502 150 2025-12-11 19:36:41,209 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b03a6fee8383e92b6f516'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:36:41,440 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b17f315f407b7629df39a 2025-12-11 19:36:41,445 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b17f315f407b7629df39a HTTP/1.1" 502 150 2025-12-11 19:36:41,445 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b17f315f407b7629df39a'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:36:43,209 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b03a6fee8383e92b6f516 2025-12-11 19:36:43,214 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b03a6fee8383e92b6f516 HTTP/1.1" 502 150 2025-12-11 19:36:43,214 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b03a6fee8383e92b6f516'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:43342 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:36:47,215 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b03a6fee8383e92b6f516 2025-12-11 19:36:47,216 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b03a6fee8383e92b6f516 HTTP/1.1" 502 150 2025-12-11 19:36:47,216 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b03a6fee8383e92b6f516'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:36:47,400 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b12c9fee8383e92b73e6d 2025-12-11 19:36:47,401 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b12c9fee8383e92b73e6d HTTP/1.1" 502 150 2025-12-11 19:36:47,401 - lava_callback - ERROR - Error processing callback for node 693b12c9fee8383e92b73e6d: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b12c9fee8383e92b73e6d (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b12c9fee8383e92b73e6d (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b12c9fee8383e92b73e6d (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:36:49,445 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b17f315f407b7629df39a 2025-12-11 19:36:49,447 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b17f315f407b7629df39a HTTP/1.1" 502 150 2025-12-11 19:36:49,447 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b17f315f407b7629df39a'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:36:53,544 - lava_callback - INFO - Received callback for node 693b186915f407b7629df45e INFO: 172.21.0.1:52818 - "POST /node/693b186915f407b7629df45e HTTP/1.0" 202 Accepted 2025-12-11 19:36:53,551 - lava_callback - INFO - Starting async job submit for node 693b186915f407b7629df45e 2025-12-11 19:36:53,573 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:36:53,613 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b186915f407b7629df45e HTTP/1.1" 502 150 2025-12-11 19:36:53,613 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b186915f407b7629df45e'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:36:53,614 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b186915f407b7629df45e 2025-12-11 19:36:53,614 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b186915f407b7629df45e HTTP/1.1" 502 150 2025-12-11 19:36:53,614 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b186915f407b7629df45e'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:36:55,216 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b03a6fee8383e92b6f516 2025-12-11 19:36:55,217 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b03a6fee8383e92b6f516 HTTP/1.1" 502 150 2025-12-11 19:36:55,217 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b03a6fee8383e92b6f516'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:36:55,615 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b186915f407b7629df45e 2025-12-11 19:36:55,616 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b186915f407b7629df45e HTTP/1.1" 502 150 2025-12-11 19:36:55,616 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b186915f407b7629df45e'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:52824 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:36:59,616 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b186915f407b7629df45e 2025-12-11 19:36:59,617 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b186915f407b7629df45e HTTP/1.1" 502 150 2025-12-11 19:36:59,617 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b186915f407b7629df45e'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:04,647 - lava_callback - INFO - Received callback for node 693b12fafee8383e92b73f59 2025-12-11 19:37:04,656 - lava_callback - INFO - Starting async job submit for node 693b12fafee8383e92b73f59 INFO: 172.21.0.1:42184 - "POST /node/693b12fafee8383e92b73f59 HTTP/1.0" 202 Accepted 2025-12-11 19:37:04,695 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:37:04,732 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b12fafee8383e92b73f59 HTTP/1.1" 502 150 2025-12-11 19:37:04,732 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b12fafee8383e92b73f59'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:04,732 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b12fafee8383e92b73f59 2025-12-11 19:37:04,733 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b12fafee8383e92b73f59 HTTP/1.1" 502 150 2025-12-11 19:37:04,733 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b12fafee8383e92b73f59'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:05,394 - lava_callback - INFO - Received callback for node 693afadafee8383e92b6d2da 2025-12-11 19:37:05,401 - lava_callback - INFO - Starting async job submit for node 693afadafee8383e92b6d2da INFO: 172.21.0.1:42198 - "POST /node/693afadafee8383e92b6d2da HTTP/1.0" 202 Accepted 2025-12-11 19:37:05,452 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b17f315f407b7629df39a 2025-12-11 19:37:05,455 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:37:05,455 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b17f315f407b7629df39a HTTP/1.1" 502 150 2025-12-11 19:37:05,455 - lava_callback - ERROR - Error processing callback for node 693b17f315f407b7629df39a: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b17f315f407b7629df39a (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b17f315f407b7629df39a (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b17f315f407b7629df39a (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:37:05,465 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afadafee8383e92b6d2da HTTP/1.1" 502 150 2025-12-11 19:37:05,465 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afadafee8383e92b6d2da'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:05,465 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afadafee8383e92b6d2da 2025-12-11 19:37:05,466 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afadafee8383e92b6d2da HTTP/1.1" 502 150 2025-12-11 19:37:05,466 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afadafee8383e92b6d2da'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:42212 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:37:06,733 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b12fafee8383e92b73f59 2025-12-11 19:37:06,734 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b12fafee8383e92b73f59 HTTP/1.1" 502 150 2025-12-11 19:37:06,734 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b12fafee8383e92b73f59'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:07,466 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afadafee8383e92b6d2da 2025-12-11 19:37:07,467 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afadafee8383e92b6d2da HTTP/1.1" 502 150 2025-12-11 19:37:07,468 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afadafee8383e92b6d2da'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:07,617 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b186915f407b7629df45e 2025-12-11 19:37:07,618 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b186915f407b7629df45e HTTP/1.1" 502 150 2025-12-11 19:37:07,618 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b186915f407b7629df45e'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:10,735 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b12fafee8383e92b73f59 2025-12-11 19:37:10,736 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b12fafee8383e92b73f59 HTTP/1.1" 502 150 2025-12-11 19:37:10,736 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b12fafee8383e92b73f59'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:10,984 - lava_callback - INFO - Received callback for node 693b17d715f407b7629df37b INFO: 172.21.0.1:47968 - "POST /node/693b17d715f407b7629df37b HTTP/1.0" 202 Accepted 2025-12-11 19:37:10,998 - lava_callback - INFO - Starting async job submit for node 693b17d715f407b7629df37b 2025-12-11 19:37:11,053 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:37:11,065 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b17d715f407b7629df37b HTTP/1.1" 502 150 2025-12-11 19:37:11,065 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b17d715f407b7629df37b'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:11,066 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b17d715f407b7629df37b 2025-12-11 19:37:11,066 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b17d715f407b7629df37b HTTP/1.1" 502 150 2025-12-11 19:37:11,066 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b17d715f407b7629df37b'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:11,218 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b03a6fee8383e92b6f516 2025-12-11 19:37:11,222 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b03a6fee8383e92b6f516 HTTP/1.1" 502 150 2025-12-11 19:37:11,222 - lava_callback - ERROR - Error processing callback for node 693b03a6fee8383e92b6f516: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b03a6fee8383e92b6f516 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b03a6fee8383e92b6f516 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b03a6fee8383e92b6f516 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:37:11,468 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afadafee8383e92b6d2da 2025-12-11 19:37:11,469 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afadafee8383e92b6d2da HTTP/1.1" 502 150 2025-12-11 19:37:11,469 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afadafee8383e92b6d2da'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:13,067 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b17d715f407b7629df37b 2025-12-11 19:37:13,068 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b17d715f407b7629df37b HTTP/1.1" 502 150 2025-12-11 19:37:13,068 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b17d715f407b7629df37b'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:47984 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 19:37:16,343 - lava_callback - INFO - Received callback for node 693af058fee8383e92b6911e INFO: 172.21.0.1:47992 - "POST /node/693af058fee8383e92b6911e HTTP/1.0" 202 Accepted 2025-12-11 19:37:16,354 - lava_callback - INFO - Starting async job submit for node 693af058fee8383e92b6911e 2025-12-11 19:37:16,392 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:37:16,428 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af058fee8383e92b6911e HTTP/1.1" 502 150 2025-12-11 19:37:16,428 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af058fee8383e92b6911e'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:16,428 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af058fee8383e92b6911e 2025-12-11 19:37:16,429 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af058fee8383e92b6911e HTTP/1.1" 502 150 2025-12-11 19:37:16,430 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af058fee8383e92b6911e'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:48004 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:37:17,068 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b17d715f407b7629df37b 2025-12-11 19:37:17,069 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b17d715f407b7629df37b HTTP/1.1" 502 150 2025-12-11 19:37:17,069 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b17d715f407b7629df37b'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:18,430 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af058fee8383e92b6911e 2025-12-11 19:37:18,431 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af058fee8383e92b6911e HTTP/1.1" 502 150 2025-12-11 19:37:18,431 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af058fee8383e92b6911e'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:18,736 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b12fafee8383e92b73f59 2025-12-11 19:37:18,737 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b12fafee8383e92b73f59 HTTP/1.1" 502 150 2025-12-11 19:37:18,737 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b12fafee8383e92b73f59'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:19,469 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afadafee8383e92b6d2da 2025-12-11 19:37:19,470 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afadafee8383e92b6d2da HTTP/1.1" 502 150 2025-12-11 19:37:19,471 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afadafee8383e92b6d2da'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:22,432 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af058fee8383e92b6911e 2025-12-11 19:37:22,433 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af058fee8383e92b6911e HTTP/1.1" 502 150 2025-12-11 19:37:22,433 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af058fee8383e92b6911e'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:23,619 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b186915f407b7629df45e 2025-12-11 19:37:23,620 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b186915f407b7629df45e HTTP/1.1" 502 150 2025-12-11 19:37:23,620 - lava_callback - ERROR - Error processing callback for node 693b186915f407b7629df45e: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b186915f407b7629df45e (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b186915f407b7629df45e (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b186915f407b7629df45e (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:37:24,796 - lava_callback - INFO - Received callback for node 693b0eebfee8383e92b7293c INFO: 172.21.0.1:35618 - "POST /node/693b0eebfee8383e92b7293c HTTP/1.0" 202 Accepted 2025-12-11 19:37:24,803 - lava_callback - INFO - Starting async job submit for node 693b0eebfee8383e92b7293c 2025-12-11 19:37:24,839 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:37:24,849 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0eebfee8383e92b7293c HTTP/1.1" 502 150 2025-12-11 19:37:24,849 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0eebfee8383e92b7293c'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:24,849 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0eebfee8383e92b7293c 2025-12-11 19:37:24,850 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0eebfee8383e92b7293c HTTP/1.1" 502 150 2025-12-11 19:37:24,850 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0eebfee8383e92b7293c'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:25,070 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b17d715f407b7629df37b 2025-12-11 19:37:25,071 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b17d715f407b7629df37b HTTP/1.1" 502 150 2025-12-11 19:37:25,071 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b17d715f407b7629df37b'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:35630 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35640 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:37:26,850 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0eebfee8383e92b7293c 2025-12-11 19:37:26,851 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0eebfee8383e92b7293c HTTP/1.1" 502 150 2025-12-11 19:37:26,851 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0eebfee8383e92b7293c'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:30,433 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af058fee8383e92b6911e 2025-12-11 19:37:30,434 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af058fee8383e92b6911e HTTP/1.1" 502 150 2025-12-11 19:37:30,434 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af058fee8383e92b6911e'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:30,851 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0eebfee8383e92b7293c 2025-12-11 19:37:30,853 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0eebfee8383e92b7293c HTTP/1.1" 502 150 2025-12-11 19:37:30,853 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0eebfee8383e92b7293c'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:34,737 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b12fafee8383e92b73f59 2025-12-11 19:37:34,739 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b12fafee8383e92b73f59 HTTP/1.1" 502 150 2025-12-11 19:37:34,739 - lava_callback - ERROR - Error processing callback for node 693b12fafee8383e92b73f59: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b12fafee8383e92b73f59 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b12fafee8383e92b73f59 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b12fafee8383e92b73f59 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:37:35,471 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afadafee8383e92b6d2da 2025-12-11 19:37:35,472 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afadafee8383e92b6d2da HTTP/1.1" 502 150 2025-12-11 19:37:35,472 - lava_callback - ERROR - Error processing callback for node 693afadafee8383e92b6d2da: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afadafee8383e92b6d2da (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afadafee8383e92b6d2da (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afadafee8383e92b6d2da (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:57770 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:37:38,853 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0eebfee8383e92b7293c 2025-12-11 19:37:38,860 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0eebfee8383e92b7293c HTTP/1.1" 502 150 2025-12-11 19:37:38,860 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0eebfee8383e92b7293c'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:41,071 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b17d715f407b7629df37b 2025-12-11 19:37:41,073 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b17d715f407b7629df37b HTTP/1.1" 502 150 2025-12-11 19:37:41,073 - lava_callback - ERROR - Error processing callback for node 693b17d715f407b7629df37b: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b17d715f407b7629df37b (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b17d715f407b7629df37b (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b17d715f407b7629df37b (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:37:46,435 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af058fee8383e92b6911e 2025-12-11 19:37:46,436 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af058fee8383e92b6911e HTTP/1.1" 502 150 2025-12-11 19:37:46,436 - lava_callback - ERROR - Error processing callback for node 693af058fee8383e92b6911e: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af058fee8383e92b6911e (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af058fee8383e92b6911e (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af058fee8383e92b6911e (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:43424 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:37:48,831 - lava_callback - INFO - Received callback for node 693b05f6fee8383e92b704e8 INFO: 172.21.0.1:43434 - "POST /node/693b05f6fee8383e92b704e8 HTTP/1.0" 202 Accepted 2025-12-11 19:37:48,848 - lava_callback - INFO - Starting async job submit for node 693b05f6fee8383e92b704e8 2025-12-11 19:37:48,950 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:37:48,983 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b05f6fee8383e92b704e8 HTTP/1.1" 502 150 2025-12-11 19:37:48,983 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b05f6fee8383e92b704e8'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:48,983 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b05f6fee8383e92b704e8 2025-12-11 19:37:48,984 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b05f6fee8383e92b704e8 HTTP/1.1" 502 150 2025-12-11 19:37:48,985 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b05f6fee8383e92b704e8'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:50,985 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b05f6fee8383e92b704e8 2025-12-11 19:37:50,986 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b05f6fee8383e92b704e8 HTTP/1.1" 502 150 2025-12-11 19:37:50,986 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b05f6fee8383e92b704e8'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:54,860 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0eebfee8383e92b7293c 2025-12-11 19:37:54,862 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0eebfee8383e92b7293c HTTP/1.1" 502 150 2025-12-11 19:37:54,862 - lava_callback - ERROR - Error processing callback for node 693b0eebfee8383e92b7293c: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0eebfee8383e92b7293c (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0eebfee8383e92b7293c (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0eebfee8383e92b7293c (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:37:54,986 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b05f6fee8383e92b704e8 2025-12-11 19:37:54,987 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b05f6fee8383e92b704e8 HTTP/1.1" 502 150 2025-12-11 19:37:54,987 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b05f6fee8383e92b704e8'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:51602 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:37:59,591 - lava_callback - INFO - Received callback for node 693b186915f407b7629df45a INFO: 172.21.0.1:51618 - "POST /node/693b186915f407b7629df45a HTTP/1.0" 202 Accepted 2025-12-11 19:37:59,597 - lava_callback - INFO - Starting async job submit for node 693b186915f407b7629df45a 2025-12-11 19:37:59,619 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:37:59,628 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b186915f407b7629df45a HTTP/1.1" 502 150 2025-12-11 19:37:59,628 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b186915f407b7629df45a'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:37:59,628 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b186915f407b7629df45a 2025-12-11 19:37:59,629 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b186915f407b7629df45a HTTP/1.1" 502 150 2025-12-11 19:37:59,629 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b186915f407b7629df45a'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:38:01,629 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b186915f407b7629df45a 2025-12-11 19:38:01,630 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b186915f407b7629df45a HTTP/1.1" 502 150 2025-12-11 19:38:01,631 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b186915f407b7629df45a'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:38:02,988 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b05f6fee8383e92b704e8 2025-12-11 19:38:02,989 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b05f6fee8383e92b704e8 HTTP/1.1" 502 150 2025-12-11 19:38:02,989 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b05f6fee8383e92b704e8'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:38:04,772 - lava_callback - INFO - Received callback for node 693af2a5fee8383e92b6a4a9 INFO: 172.21.0.1:39092 - "POST /node/693af2a5fee8383e92b6a4a9 HTTP/1.0" 202 Accepted 2025-12-11 19:38:04,782 - lava_callback - INFO - Starting async job submit for node 693af2a5fee8383e92b6a4a9 2025-12-11 19:38:04,809 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:38:04,822 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af2a5fee8383e92b6a4a9 HTTP/1.1" 502 150 2025-12-11 19:38:04,822 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af2a5fee8383e92b6a4a9'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:38:04,823 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af2a5fee8383e92b6a4a9 2025-12-11 19:38:04,824 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af2a5fee8383e92b6a4a9 HTTP/1.1" 502 150 2025-12-11 19:38:04,824 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af2a5fee8383e92b6a4a9'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:38:05,544 - lava_callback - INFO - Received callback for node 693ae84bfee8383e92b6587f INFO: 172.21.0.1:39094 - "POST /node/693ae84bfee8383e92b6587f HTTP/1.0" 202 Accepted 2025-12-11 19:38:05,554 - lava_callback - INFO - Starting async job submit for node 693ae84bfee8383e92b6587f 2025-12-11 19:38:05,579 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:38:05,595 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae84bfee8383e92b6587f HTTP/1.1" 502 150 2025-12-11 19:38:05,595 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae84bfee8383e92b6587f'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:38:05,595 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae84bfee8383e92b6587f 2025-12-11 19:38:05,595 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae84bfee8383e92b6587f HTTP/1.1" 502 150 2025-12-11 19:38:05,596 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae84bfee8383e92b6587f'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:38:05,631 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b186915f407b7629df45a 2025-12-11 19:38:05,632 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b186915f407b7629df45a HTTP/1.1" 502 150 2025-12-11 19:38:05,632 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b186915f407b7629df45a'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:39106 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:38:06,824 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af2a5fee8383e92b6a4a9 2025-12-11 19:38:06,825 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af2a5fee8383e92b6a4a9 HTTP/1.1" 502 150 2025-12-11 19:38:06,825 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af2a5fee8383e92b6a4a9'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:38:07,596 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae84bfee8383e92b6587f 2025-12-11 19:38:07,603 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae84bfee8383e92b6587f HTTP/1.1" 502 150 2025-12-11 19:38:07,603 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae84bfee8383e92b6587f'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:38:10,825 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af2a5fee8383e92b6a4a9 2025-12-11 19:38:10,827 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af2a5fee8383e92b6a4a9 HTTP/1.1" 502 150 2025-12-11 19:38:10,827 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af2a5fee8383e92b6a4a9'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:38:11,603 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae84bfee8383e92b6587f 2025-12-11 19:38:11,605 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae84bfee8383e92b6587f HTTP/1.1" 502 150 2025-12-11 19:38:11,605 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae84bfee8383e92b6587f'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:38:13,632 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b186915f407b7629df45a 2025-12-11 19:38:13,633 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b186915f407b7629df45a HTTP/1.1" 502 150 2025-12-11 19:38:13,633 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b186915f407b7629df45a'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:59644 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:59654 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:38:17,791 - lava_callback - INFO - Received callback for node 693b12f2fee8383e92b73f38 INFO: 172.21.0.1:59666 - "POST /node/693b12f2fee8383e92b73f38 HTTP/1.0" 202 Accepted 2025-12-11 19:38:17,798 - lava_callback - INFO - Starting async job submit for node 693b12f2fee8383e92b73f38 2025-12-11 19:38:17,828 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:38:17,840 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b12f2fee8383e92b73f38 HTTP/1.1" 502 150 2025-12-11 19:38:17,840 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b12f2fee8383e92b73f38'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:38:17,840 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b12f2fee8383e92b73f38 2025-12-11 19:38:17,841 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b12f2fee8383e92b73f38 HTTP/1.1" 502 150 2025-12-11 19:38:17,841 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b12f2fee8383e92b73f38'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:38:18,827 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af2a5fee8383e92b6a4a9 2025-12-11 19:38:18,828 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af2a5fee8383e92b6a4a9 HTTP/1.1" 502 150 2025-12-11 19:38:18,828 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af2a5fee8383e92b6a4a9'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:38:18,989 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b05f6fee8383e92b704e8 2025-12-11 19:38:18,991 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b05f6fee8383e92b704e8 HTTP/1.1" 502 150 2025-12-11 19:38:18,991 - lava_callback - ERROR - Error processing callback for node 693b05f6fee8383e92b704e8: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b05f6fee8383e92b704e8 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b05f6fee8383e92b704e8 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b05f6fee8383e92b704e8 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:38:19,605 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae84bfee8383e92b6587f 2025-12-11 19:38:19,606 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae84bfee8383e92b6587f HTTP/1.1" 502 150 2025-12-11 19:38:19,606 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae84bfee8383e92b6587f'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:38:19,841 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b12f2fee8383e92b73f38 2025-12-11 19:38:19,842 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b12f2fee8383e92b73f38 HTTP/1.1" 502 150 2025-12-11 19:38:19,842 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b12f2fee8383e92b73f38'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:38:23,842 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b12f2fee8383e92b73f38 2025-12-11 19:38:23,844 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b12f2fee8383e92b73f38 HTTP/1.1" 502 150 2025-12-11 19:38:23,844 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b12f2fee8383e92b73f38'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:53288 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53300 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:38:29,634 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b186915f407b7629df45a 2025-12-11 19:38:29,635 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b186915f407b7629df45a HTTP/1.1" 502 150 2025-12-11 19:38:29,635 - lava_callback - ERROR - Error processing callback for node 693b186915f407b7629df45a: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b186915f407b7629df45a (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b186915f407b7629df45a (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b186915f407b7629df45a (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:38:31,844 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b12f2fee8383e92b73f38 2025-12-11 19:38:31,845 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b12f2fee8383e92b73f38 HTTP/1.1" 502 150 2025-12-11 19:38:31,845 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b12f2fee8383e92b73f38'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:38:33,576 - lava_callback - INFO - Received callback for node 693b186915f407b7629df45c INFO: 172.21.0.1:42528 - "POST /node/693b186915f407b7629df45c HTTP/1.0" 202 Accepted 2025-12-11 19:38:33,583 - lava_callback - INFO - Starting async job submit for node 693b186915f407b7629df45c 2025-12-11 19:38:33,628 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:38:33,638 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b186915f407b7629df45c HTTP/1.1" 502 150 2025-12-11 19:38:33,638 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b186915f407b7629df45c'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:38:33,638 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b186915f407b7629df45c 2025-12-11 19:38:33,639 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b186915f407b7629df45c HTTP/1.1" 502 150 2025-12-11 19:38:33,639 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b186915f407b7629df45c'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:38:34,828 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af2a5fee8383e92b6a4a9 2025-12-11 19:38:34,830 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af2a5fee8383e92b6a4a9 HTTP/1.1" 502 150 2025-12-11 19:38:34,830 - lava_callback - ERROR - Error processing callback for node 693af2a5fee8383e92b6a4a9: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af2a5fee8383e92b6a4a9 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af2a5fee8383e92b6a4a9 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af2a5fee8383e92b6a4a9 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:38:35,606 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae84bfee8383e92b6587f 2025-12-11 19:38:35,607 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae84bfee8383e92b6587f HTTP/1.1" 502 150 2025-12-11 19:38:35,608 - lava_callback - ERROR - Error processing callback for node 693ae84bfee8383e92b6587f: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae84bfee8383e92b6587f (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae84bfee8383e92b6587f (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae84bfee8383e92b6587f (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:38:35,639 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b186915f407b7629df45c 2025-12-11 19:38:35,643 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b186915f407b7629df45c HTTP/1.1" 502 150 2025-12-11 19:38:35,643 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b186915f407b7629df45c'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:42536 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:38:39,643 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b186915f407b7629df45c 2025-12-11 19:38:39,645 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b186915f407b7629df45c HTTP/1.1" 502 150 2025-12-11 19:38:39,645 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b186915f407b7629df45c'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:43174 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:38:47,645 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b186915f407b7629df45c 2025-12-11 19:38:47,646 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b186915f407b7629df45c HTTP/1.1" 502 150 2025-12-11 19:38:47,646 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b186915f407b7629df45c'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:38:47,845 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b12f2fee8383e92b73f38 2025-12-11 19:38:47,846 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b12f2fee8383e92b73f38 HTTP/1.1" 502 150 2025-12-11 19:38:47,847 - lava_callback - ERROR - Error processing callback for node 693b12f2fee8383e92b73f38: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b12f2fee8383e92b73f38 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b12f2fee8383e92b73f38 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b12f2fee8383e92b73f38 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:38:48,770 - lava_callback - INFO - Received callback for node 693b0466fee8383e92b6f83f INFO: 172.21.0.1:43184 - "POST /node/693b0466fee8383e92b6f83f HTTP/1.0" 202 Accepted 2025-12-11 19:38:48,779 - lava_callback - INFO - Starting async job submit for node 693b0466fee8383e92b6f83f 2025-12-11 19:38:48,851 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:38:48,861 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0466fee8383e92b6f83f HTTP/1.1" 502 150 2025-12-11 19:38:48,861 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0466fee8383e92b6f83f'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:38:48,862 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0466fee8383e92b6f83f 2025-12-11 19:38:48,862 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0466fee8383e92b6f83f HTTP/1.1" 502 150 2025-12-11 19:38:48,862 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0466fee8383e92b6f83f'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:38:50,863 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0466fee8383e92b6f83f 2025-12-11 19:38:50,864 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0466fee8383e92b6f83f HTTP/1.1" 502 150 2025-12-11 19:38:50,864 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0466fee8383e92b6f83f'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:38:54,864 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0466fee8383e92b6f83f 2025-12-11 19:38:54,866 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0466fee8383e92b6f83f HTTP/1.1" 502 150 2025-12-11 19:38:54,866 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0466fee8383e92b6f83f'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:36622 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:39:02,866 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0466fee8383e92b6f83f 2025-12-11 19:39:02,867 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0466fee8383e92b6f83f HTTP/1.1" 502 150 2025-12-11 19:39:02,867 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0466fee8383e92b6f83f'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:03,138 - lava_callback - INFO - Received callback for node 693b0418fee8383e92b6f724 INFO: 172.21.0.1:53650 - "POST /node/693b0418fee8383e92b6f724 HTTP/1.0" 202 Accepted 2025-12-11 19:39:03,147 - lava_callback - INFO - Starting async job submit for node 693b0418fee8383e92b6f724 2025-12-11 19:39:03,206 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:39:03,223 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0418fee8383e92b6f724 HTTP/1.1" 502 150 2025-12-11 19:39:03,223 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0418fee8383e92b6f724'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:03,224 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0418fee8383e92b6f724 2025-12-11 19:39:03,224 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0418fee8383e92b6f724 HTTP/1.1" 502 150 2025-12-11 19:39:03,224 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0418fee8383e92b6f724'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:03,646 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b186915f407b7629df45c 2025-12-11 19:39:03,648 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b186915f407b7629df45c HTTP/1.1" 502 150 2025-12-11 19:39:03,648 - lava_callback - ERROR - Error processing callback for node 693b186915f407b7629df45c: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b186915f407b7629df45c (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b186915f407b7629df45c (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b186915f407b7629df45c (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:39:04,324 - lava_callback - INFO - Received callback for node 693b061efee8383e92b70561 INFO: 172.21.0.1:53658 - "POST /node/693b061efee8383e92b70561 HTTP/1.0" 202 Accepted 2025-12-11 19:39:04,332 - lava_callback - INFO - Starting async job submit for node 693b061efee8383e92b70561 2025-12-11 19:39:04,371 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:39:04,386 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b061efee8383e92b70561 HTTP/1.1" 502 150 2025-12-11 19:39:04,386 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b061efee8383e92b70561'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:04,386 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b061efee8383e92b70561 2025-12-11 19:39:04,387 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b061efee8383e92b70561 HTTP/1.1" 502 150 2025-12-11 19:39:04,387 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b061efee8383e92b70561'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:05,225 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0418fee8383e92b6f724 2025-12-11 19:39:05,226 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0418fee8383e92b6f724 HTTP/1.1" 502 150 2025-12-11 19:39:05,226 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0418fee8383e92b6f724'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:06,387 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b061efee8383e92b70561 2025-12-11 19:39:06,388 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b061efee8383e92b70561 HTTP/1.1" 502 150 2025-12-11 19:39:06,388 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b061efee8383e92b70561'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:53660 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:39:07,437 - lava_callback - INFO - Received callback for node 693af059fee8383e92b69126 INFO: 172.21.0.1:53668 - "POST /node/693af059fee8383e92b69126 HTTP/1.0" 202 Accepted 2025-12-11 19:39:07,447 - lava_callback - INFO - Starting async job submit for node 693af059fee8383e92b69126 2025-12-11 19:39:07,485 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:39:07,495 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af059fee8383e92b69126 HTTP/1.1" 502 150 2025-12-11 19:39:07,496 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af059fee8383e92b69126'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:07,496 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af059fee8383e92b69126 2025-12-11 19:39:07,496 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af059fee8383e92b69126 HTTP/1.1" 502 150 2025-12-11 19:39:07,496 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af059fee8383e92b69126'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:09,226 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0418fee8383e92b6f724 2025-12-11 19:39:09,227 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0418fee8383e92b6f724 HTTP/1.1" 502 150 2025-12-11 19:39:09,227 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0418fee8383e92b6f724'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:09,497 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af059fee8383e92b69126 2025-12-11 19:39:09,498 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af059fee8383e92b69126 HTTP/1.1" 502 150 2025-12-11 19:39:09,498 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af059fee8383e92b69126'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:10,389 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b061efee8383e92b70561 2025-12-11 19:39:10,390 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b061efee8383e92b70561 HTTP/1.1" 502 150 2025-12-11 19:39:10,390 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b061efee8383e92b70561'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:13,498 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af059fee8383e92b69126 2025-12-11 19:39:13,499 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af059fee8383e92b69126 HTTP/1.1" 502 150 2025-12-11 19:39:13,499 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af059fee8383e92b69126'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:40478 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:40484 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:39:17,228 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0418fee8383e92b6f724 2025-12-11 19:39:17,229 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0418fee8383e92b6f724 HTTP/1.1" 502 150 2025-12-11 19:39:17,229 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0418fee8383e92b6f724'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:17,961 - lava_callback - INFO - Received callback for node 693b189315f407b7629df48b INFO: 172.21.0.1:40496 - "POST /node/693b189315f407b7629df48b HTTP/1.0" 202 Accepted 2025-12-11 19:39:17,968 - lava_callback - INFO - Starting async job submit for node 693b189315f407b7629df48b 2025-12-11 19:39:18,004 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:39:18,016 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b189315f407b7629df48b HTTP/1.1" 502 150 2025-12-11 19:39:18,016 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b189315f407b7629df48b'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:18,016 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b189315f407b7629df48b 2025-12-11 19:39:18,017 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b189315f407b7629df48b HTTP/1.1" 502 150 2025-12-11 19:39:18,017 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b189315f407b7629df48b'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:18,390 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b061efee8383e92b70561 2025-12-11 19:39:18,392 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b061efee8383e92b70561 HTTP/1.1" 502 150 2025-12-11 19:39:18,392 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b061efee8383e92b70561'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:18,867 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0466fee8383e92b6f83f 2025-12-11 19:39:18,869 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0466fee8383e92b6f83f HTTP/1.1" 502 150 2025-12-11 19:39:18,869 - lava_callback - ERROR - Error processing callback for node 693b0466fee8383e92b6f83f: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0466fee8383e92b6f83f (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0466fee8383e92b6f83f (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0466fee8383e92b6f83f (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:39:20,017 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b189315f407b7629df48b 2025-12-11 19:39:20,018 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b189315f407b7629df48b HTTP/1.1" 502 150 2025-12-11 19:39:20,018 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b189315f407b7629df48b'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:21,500 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af059fee8383e92b69126 2025-12-11 19:39:21,501 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af059fee8383e92b69126 HTTP/1.1" 502 150 2025-12-11 19:39:21,501 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af059fee8383e92b69126'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:24,018 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b189315f407b7629df48b 2025-12-11 19:39:24,019 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b189315f407b7629df48b HTTP/1.1" 502 150 2025-12-11 19:39:24,020 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b189315f407b7629df48b'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:25,612 - lava_callback - INFO - Received callback for node 693ae846fee8383e92b6586e INFO: 172.21.0.1:38768 - "POST /node/693ae846fee8383e92b6586e HTTP/1.0" 202 Accepted 2025-12-11 19:39:25,619 - lava_callback - INFO - Starting async job submit for node 693ae846fee8383e92b6586e 2025-12-11 19:39:25,733 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:39:25,744 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae846fee8383e92b6586e HTTP/1.1" 502 150 2025-12-11 19:39:25,744 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae846fee8383e92b6586e'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:25,745 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae846fee8383e92b6586e 2025-12-11 19:39:25,746 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae846fee8383e92b6586e HTTP/1.1" 502 150 2025-12-11 19:39:25,746 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae846fee8383e92b6586e'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:38774 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38786 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:39:27,528 - lava_callback - INFO - Received callback for node 6938868c63d980a974804883 2025-12-11 19:39:27,539 - lava_callback - INFO - Starting async job submit for node 6938868c63d980a974804883 INFO: 172.21.0.1:38792 - "POST /node/6938868c63d980a974804883 HTTP/1.0" 202 Accepted 2025-12-11 19:39:27,594 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:39:27,617 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938868c63d980a974804883 HTTP/1.1" 502 150 2025-12-11 19:39:27,617 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938868c63d980a974804883'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:27,618 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938868c63d980a974804883 2025-12-11 19:39:27,618 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938868c63d980a974804883 HTTP/1.1" 502 150 2025-12-11 19:39:27,619 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938868c63d980a974804883'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:27,746 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae846fee8383e92b6586e 2025-12-11 19:39:27,747 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae846fee8383e92b6586e HTTP/1.1" 502 150 2025-12-11 19:39:27,747 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae846fee8383e92b6586e'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:29,619 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938868c63d980a974804883 2025-12-11 19:39:29,620 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938868c63d980a974804883 HTTP/1.1" 502 150 2025-12-11 19:39:29,620 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938868c63d980a974804883'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:31,748 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae846fee8383e92b6586e 2025-12-11 19:39:31,749 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae846fee8383e92b6586e HTTP/1.1" 502 150 2025-12-11 19:39:31,749 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae846fee8383e92b6586e'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:32,020 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b189315f407b7629df48b 2025-12-11 19:39:32,022 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b189315f407b7629df48b HTTP/1.1" 502 150 2025-12-11 19:39:32,022 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b189315f407b7629df48b'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:33,229 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0418fee8383e92b6f724 2025-12-11 19:39:33,231 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0418fee8383e92b6f724 HTTP/1.1" 502 150 2025-12-11 19:39:33,231 - lava_callback - ERROR - Error processing callback for node 693b0418fee8383e92b6f724: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0418fee8383e92b6f724 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0418fee8383e92b6f724 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0418fee8383e92b6f724 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:39:33,620 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938868c63d980a974804883 2025-12-11 19:39:33,622 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938868c63d980a974804883 HTTP/1.1" 502 150 2025-12-11 19:39:33,622 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938868c63d980a974804883'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:34,392 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b061efee8383e92b70561 2025-12-11 19:39:34,394 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b061efee8383e92b70561 HTTP/1.1" 502 150 2025-12-11 19:39:34,394 - lava_callback - ERROR - Error processing callback for node 693b061efee8383e92b70561: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b061efee8383e92b70561 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b061efee8383e92b70561 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b061efee8383e92b70561 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:39344 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:39:37,501 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af059fee8383e92b69126 2025-12-11 19:39:37,509 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af059fee8383e92b69126 HTTP/1.1" 502 150 2025-12-11 19:39:37,509 - lava_callback - ERROR - Error processing callback for node 693af059fee8383e92b69126: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af059fee8383e92b69126 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af059fee8383e92b69126 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af059fee8383e92b69126 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:39:39,749 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae846fee8383e92b6586e 2025-12-11 19:39:39,750 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae846fee8383e92b6586e HTTP/1.1" 502 150 2025-12-11 19:39:39,751 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae846fee8383e92b6586e'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:41,622 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938868c63d980a974804883 2025-12-11 19:39:41,623 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938868c63d980a974804883 HTTP/1.1" 502 150 2025-12-11 19:39:41,623 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938868c63d980a974804883'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:44,805 - lava_callback - INFO - Received callback for node 693b116ffee8383e92b73897 INFO: 172.21.0.1:56216 - "POST /node/693b116ffee8383e92b73897 HTTP/1.0" 202 Accepted 2025-12-11 19:39:44,811 - lava_callback - INFO - Starting async job submit for node 693b116ffee8383e92b73897 2025-12-11 19:39:44,834 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:39:44,844 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b116ffee8383e92b73897 HTTP/1.1" 502 150 2025-12-11 19:39:44,844 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b116ffee8383e92b73897'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:44,844 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b116ffee8383e92b73897 2025-12-11 19:39:44,844 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b116ffee8383e92b73897 HTTP/1.1" 502 150 2025-12-11 19:39:44,844 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b116ffee8383e92b73897'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:46,304 - lava_callback - INFO - Received callback for node 693b1343fee8383e92b74097 INFO: 172.21.0.1:56228 - "POST /node/693b1343fee8383e92b74097 HTTP/1.0" 202 Accepted 2025-12-11 19:39:46,311 - lava_callback - INFO - Starting async job submit for node 693b1343fee8383e92b74097 2025-12-11 19:39:46,347 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:39:46,357 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1343fee8383e92b74097 HTTP/1.1" 502 150 2025-12-11 19:39:46,358 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1343fee8383e92b74097'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:46,358 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1343fee8383e92b74097 2025-12-11 19:39:46,358 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1343fee8383e92b74097 HTTP/1.1" 502 150 2025-12-11 19:39:46,359 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1343fee8383e92b74097'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:56240 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:39:46,845 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b116ffee8383e92b73897 2025-12-11 19:39:46,846 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b116ffee8383e92b73897 HTTP/1.1" 502 150 2025-12-11 19:39:46,846 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b116ffee8383e92b73897'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:48,023 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b189315f407b7629df48b 2025-12-11 19:39:48,024 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b189315f407b7629df48b HTTP/1.1" 502 150 2025-12-11 19:39:48,024 - lava_callback - ERROR - Error processing callback for node 693b189315f407b7629df48b: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b189315f407b7629df48b (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b189315f407b7629df48b (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b189315f407b7629df48b (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:39:48,359 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1343fee8383e92b74097 2025-12-11 19:39:48,360 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1343fee8383e92b74097 HTTP/1.1" 502 150 2025-12-11 19:39:48,360 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1343fee8383e92b74097'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:50,846 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b116ffee8383e92b73897 2025-12-11 19:39:50,847 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b116ffee8383e92b73897 HTTP/1.1" 502 150 2025-12-11 19:39:50,847 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b116ffee8383e92b73897'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:52,360 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1343fee8383e92b74097 2025-12-11 19:39:52,362 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1343fee8383e92b74097 HTTP/1.1" 502 150 2025-12-11 19:39:52,362 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1343fee8383e92b74097'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:39:55,751 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae846fee8383e92b6586e 2025-12-11 19:39:55,753 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae846fee8383e92b6586e HTTP/1.1" 502 150 2025-12-11 19:39:55,753 - lava_callback - ERROR - Error processing callback for node 693ae846fee8383e92b6586e: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae846fee8383e92b6586e (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae846fee8383e92b6586e (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae846fee8383e92b6586e (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:41332 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:39:57,623 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938868c63d980a974804883 2025-12-11 19:39:57,624 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938868c63d980a974804883 HTTP/1.1" 502 150 2025-12-11 19:39:57,625 - lava_callback - ERROR - Error processing callback for node 6938868c63d980a974804883: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938868c63d980a974804883 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938868c63d980a974804883 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938868c63d980a974804883 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:39:58,847 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b116ffee8383e92b73897 2025-12-11 19:39:58,848 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b116ffee8383e92b73897 HTTP/1.1" 502 150 2025-12-11 19:39:58,849 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b116ffee8383e92b73897'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:40:00,362 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1343fee8383e92b74097 2025-12-11 19:40:00,363 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1343fee8383e92b74097 HTTP/1.1" 502 150 2025-12-11 19:40:00,364 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1343fee8383e92b74097'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:40:05,541 - lava_callback - INFO - Received callback for node 693ae83dfee8383e92b65847 INFO: 172.21.0.1:60462 - "POST /node/693ae83dfee8383e92b65847 HTTP/1.0" 202 Accepted 2025-12-11 19:40:05,589 - lava_callback - INFO - Starting async job submit for node 693ae83dfee8383e92b65847 2025-12-11 19:40:05,615 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:40:05,668 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae83dfee8383e92b65847 HTTP/1.1" 502 150 2025-12-11 19:40:05,668 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae83dfee8383e92b65847'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:40:05,668 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae83dfee8383e92b65847 2025-12-11 19:40:05,669 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae83dfee8383e92b65847 HTTP/1.1" 502 150 2025-12-11 19:40:05,670 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae83dfee8383e92b65847'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:60476 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:40:07,670 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae83dfee8383e92b65847 2025-12-11 19:40:07,671 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae83dfee8383e92b65847 HTTP/1.1" 502 150 2025-12-11 19:40:07,671 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae83dfee8383e92b65847'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:40:11,671 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae83dfee8383e92b65847 2025-12-11 19:40:11,672 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae83dfee8383e92b65847 HTTP/1.1" 502 150 2025-12-11 19:40:11,672 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae83dfee8383e92b65847'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:40:14,822 - lava_callback - INFO - Received callback for node 693b189315f407b7629df48f INFO: 172.21.0.1:52992 - "POST /node/693b189315f407b7629df48f HTTP/1.0" 202 Accepted 2025-12-11 19:40:14,830 - lava_callback - INFO - Starting async job submit for node 693b189315f407b7629df48f 2025-12-11 19:40:14,854 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b116ffee8383e92b73897 2025-12-11 19:40:14,855 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:40:14,855 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b116ffee8383e92b73897 HTTP/1.1" 502 150 2025-12-11 19:40:14,856 - lava_callback - ERROR - Error processing callback for node 693b116ffee8383e92b73897: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b116ffee8383e92b73897 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b116ffee8383e92b73897 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b116ffee8383e92b73897 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:40:14,865 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b189315f407b7629df48f HTTP/1.1" 502 150 2025-12-11 19:40:14,865 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b189315f407b7629df48f'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:40:14,865 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b189315f407b7629df48f 2025-12-11 19:40:14,871 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b189315f407b7629df48f HTTP/1.1" 502 150 2025-12-11 19:40:14,871 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b189315f407b7629df48f'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:53008 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 19:40:16,364 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1343fee8383e92b74097 2025-12-11 19:40:16,365 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1343fee8383e92b74097 HTTP/1.1" 502 150 2025-12-11 19:40:16,365 - lava_callback - ERROR - Error processing callback for node 693b1343fee8383e92b74097: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1343fee8383e92b74097 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1343fee8383e92b74097 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1343fee8383e92b74097 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:53020 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:40:16,827 - lava_callback - INFO - Received callback for node 693b134dfee8383e92b740c0 INFO: 172.21.0.1:53036 - "POST /node/693b134dfee8383e92b740c0 HTTP/1.0" 202 Accepted 2025-12-11 19:40:16,833 - lava_callback - INFO - Starting async job submit for node 693b134dfee8383e92b740c0 2025-12-11 19:40:16,858 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:40:16,870 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b134dfee8383e92b740c0 HTTP/1.1" 502 150 2025-12-11 19:40:16,870 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b134dfee8383e92b740c0'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:40:16,870 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b134dfee8383e92b740c0 2025-12-11 19:40:16,871 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b134dfee8383e92b740c0 HTTP/1.1" 502 150 2025-12-11 19:40:16,871 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b134dfee8383e92b740c0'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:40:16,871 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b189315f407b7629df48f 2025-12-11 19:40:16,872 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b189315f407b7629df48f HTTP/1.1" 502 150 2025-12-11 19:40:16,872 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b189315f407b7629df48f'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:40:18,871 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b134dfee8383e92b740c0 2025-12-11 19:40:18,872 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b134dfee8383e92b740c0 HTTP/1.1" 502 150 2025-12-11 19:40:18,872 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b134dfee8383e92b740c0'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:40:19,673 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae83dfee8383e92b65847 2025-12-11 19:40:19,674 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae83dfee8383e92b65847 HTTP/1.1" 502 150 2025-12-11 19:40:19,674 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae83dfee8383e92b65847'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:40:20,872 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b189315f407b7629df48f 2025-12-11 19:40:20,873 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b189315f407b7629df48f HTTP/1.1" 502 150 2025-12-11 19:40:20,873 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b189315f407b7629df48f'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:40:22,873 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b134dfee8383e92b740c0 2025-12-11 19:40:22,874 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b134dfee8383e92b740c0 HTTP/1.1" 502 150 2025-12-11 19:40:22,874 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b134dfee8383e92b740c0'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:38918 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38926 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:40:28,873 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b189315f407b7629df48f 2025-12-11 19:40:28,875 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b189315f407b7629df48f HTTP/1.1" 502 150 2025-12-11 19:40:28,875 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b189315f407b7629df48f'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:40:30,874 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b134dfee8383e92b740c0 2025-12-11 19:40:30,875 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b134dfee8383e92b740c0 HTTP/1.1" 502 150 2025-12-11 19:40:30,875 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b134dfee8383e92b740c0'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:40:35,674 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae83dfee8383e92b65847 2025-12-11 19:40:35,675 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae83dfee8383e92b65847 HTTP/1.1" 502 150 2025-12-11 19:40:35,676 - lava_callback - ERROR - Error processing callback for node 693ae83dfee8383e92b65847: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae83dfee8383e92b65847 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae83dfee8383e92b65847 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae83dfee8383e92b65847 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:56088 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:40:37,533 - lava_callback - INFO - Received callback for node 693b189315f407b7629df48d INFO: 172.21.0.1:56104 - "POST /node/693b189315f407b7629df48d HTTP/1.0" 202 Accepted 2025-12-11 19:40:37,540 - lava_callback - INFO - Starting async job submit for node 693b189315f407b7629df48d 2025-12-11 19:40:37,565 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:40:37,586 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b189315f407b7629df48d HTTP/1.1" 502 150 2025-12-11 19:40:37,586 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b189315f407b7629df48d'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:40:37,587 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b189315f407b7629df48d 2025-12-11 19:40:37,587 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b189315f407b7629df48d HTTP/1.1" 502 150 2025-12-11 19:40:37,588 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b189315f407b7629df48d'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:40:39,588 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b189315f407b7629df48d 2025-12-11 19:40:39,589 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b189315f407b7629df48d HTTP/1.1" 502 150 2025-12-11 19:40:39,589 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b189315f407b7629df48d'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:40:43,589 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b189315f407b7629df48d 2025-12-11 19:40:43,591 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b189315f407b7629df48d HTTP/1.1" 502 150 2025-12-11 19:40:43,591 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b189315f407b7629df48d'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:40:44,875 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b189315f407b7629df48f 2025-12-11 19:40:44,876 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b189315f407b7629df48f HTTP/1.1" 502 150 2025-12-11 19:40:44,876 - lava_callback - ERROR - Error processing callback for node 693b189315f407b7629df48f: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b189315f407b7629df48f (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b189315f407b7629df48f (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b189315f407b7629df48f (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:40:46,490 - lava_callback - INFO - Received callback for node 693af2d5fee8383e92b6a631 INFO: 172.21.0.1:57330 - "POST /node/693af2d5fee8383e92b6a631 HTTP/1.0" 202 Accepted 2025-12-11 19:40:46,496 - lava_callback - INFO - Starting async job submit for node 693af2d5fee8383e92b6a631 2025-12-11 19:40:46,520 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 INFO: 172.21.0.1:57338 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:40:46,529 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af2d5fee8383e92b6a631 HTTP/1.1" 502 150 2025-12-11 19:40:46,529 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af2d5fee8383e92b6a631'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:40:46,530 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af2d5fee8383e92b6a631 2025-12-11 19:40:46,530 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af2d5fee8383e92b6a631 HTTP/1.1" 502 150 2025-12-11 19:40:46,530 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af2d5fee8383e92b6a631'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:40:46,876 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b134dfee8383e92b740c0 2025-12-11 19:40:46,877 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b134dfee8383e92b740c0 HTTP/1.1" 502 150 2025-12-11 19:40:46,877 - lava_callback - ERROR - Error processing callback for node 693b134dfee8383e92b740c0: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b134dfee8383e92b740c0 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b134dfee8383e92b740c0 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b134dfee8383e92b740c0 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:40:48,530 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af2d5fee8383e92b6a631 2025-12-11 19:40:48,532 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af2d5fee8383e92b6a631 HTTP/1.1" 502 150 2025-12-11 19:40:48,532 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af2d5fee8383e92b6a631'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:40:49,954 - lava_callback - INFO - Received callback for node 693b07c1fee8383e92b7100b INFO: 172.21.0.1:41146 - "POST /node/693b07c1fee8383e92b7100b HTTP/1.0" 202 Accepted 2025-12-11 19:40:49,966 - lava_callback - INFO - Starting async job submit for node 693b07c1fee8383e92b7100b 2025-12-11 19:40:50,006 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:40:50,019 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07c1fee8383e92b7100b HTTP/1.1" 502 150 2025-12-11 19:40:50,020 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07c1fee8383e92b7100b'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:40:50,020 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07c1fee8383e92b7100b 2025-12-11 19:40:50,021 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07c1fee8383e92b7100b HTTP/1.1" 502 150 2025-12-11 19:40:50,021 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07c1fee8383e92b7100b'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:40:51,591 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b189315f407b7629df48d 2025-12-11 19:40:51,592 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b189315f407b7629df48d HTTP/1.1" 502 150 2025-12-11 19:40:51,592 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b189315f407b7629df48d'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:40:52,021 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07c1fee8383e92b7100b 2025-12-11 19:40:52,022 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07c1fee8383e92b7100b HTTP/1.1" 502 150 2025-12-11 19:40:52,022 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07c1fee8383e92b7100b'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:40:52,532 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af2d5fee8383e92b6a631 2025-12-11 19:40:52,533 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af2d5fee8383e92b6a631 HTTP/1.1" 502 150 2025-12-11 19:40:52,533 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af2d5fee8383e92b6a631'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:40:56,022 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07c1fee8383e92b7100b 2025-12-11 19:40:56,023 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07c1fee8383e92b7100b HTTP/1.1" 502 150 2025-12-11 19:40:56,023 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07c1fee8383e92b7100b'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:41160 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:41:00,533 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af2d5fee8383e92b6a631 2025-12-11 19:41:00,534 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af2d5fee8383e92b6a631 HTTP/1.1" 502 150 2025-12-11 19:41:00,534 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af2d5fee8383e92b6a631'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:41:04,024 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07c1fee8383e92b7100b 2025-12-11 19:41:04,025 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07c1fee8383e92b7100b HTTP/1.1" 502 150 2025-12-11 19:41:04,025 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07c1fee8383e92b7100b'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:40736 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:41:07,593 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b189315f407b7629df48d 2025-12-11 19:41:07,594 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b189315f407b7629df48d HTTP/1.1" 502 150 2025-12-11 19:41:07,595 - lava_callback - ERROR - Error processing callback for node 693b189315f407b7629df48d: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b189315f407b7629df48d (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b189315f407b7629df48d (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b189315f407b7629df48d (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:59764 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 19:41:15,279 - lava_callback - INFO - Received callback for node 693b18e115f407b7629df580 INFO: 172.21.0.1:59780 - "POST /node/693b18e115f407b7629df580 HTTP/1.0" 202 Accepted 2025-12-11 19:41:15,286 - lava_callback - INFO - Starting async job submit for node 693b18e115f407b7629df580 2025-12-11 19:41:15,312 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:41:15,324 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18e115f407b7629df580 HTTP/1.1" 502 150 2025-12-11 19:41:15,324 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18e115f407b7629df580'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:41:15,324 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18e115f407b7629df580 2025-12-11 19:41:15,325 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18e115f407b7629df580 HTTP/1.1" 502 150 2025-12-11 19:41:15,325 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18e115f407b7629df580'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:41:16,535 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af2d5fee8383e92b6a631 2025-12-11 19:41:16,536 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af2d5fee8383e92b6a631 HTTP/1.1" 502 150 INFO: 172.21.0.1:59782 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:41:16,536 - lava_callback - ERROR - Error processing callback for node 693af2d5fee8383e92b6a631: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af2d5fee8383e92b6a631 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af2d5fee8383e92b6a631 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af2d5fee8383e92b6a631 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:41:17,325 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18e115f407b7629df580 2025-12-11 19:41:17,326 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18e115f407b7629df580 HTTP/1.1" 502 150 2025-12-11 19:41:17,326 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18e115f407b7629df580'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:41:20,025 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07c1fee8383e92b7100b 2025-12-11 19:41:20,027 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07c1fee8383e92b7100b HTTP/1.1" 502 150 2025-12-11 19:41:20,027 - lava_callback - ERROR - Error processing callback for node 693b07c1fee8383e92b7100b: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b07c1fee8383e92b7100b (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b07c1fee8383e92b7100b (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b07c1fee8383e92b7100b (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:41:21,326 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18e115f407b7629df580 2025-12-11 19:41:21,328 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18e115f407b7629df580 HTTP/1.1" 502 150 2025-12-11 19:41:21,328 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18e115f407b7629df580'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:41:24,956 - lava_callback - INFO - Received callback for node 693af2e8fee8383e92b6a6bd INFO: 172.21.0.1:51042 - "POST /node/693af2e8fee8383e92b6a6bd HTTP/1.0" 202 Accepted 2025-12-11 19:41:24,963 - lava_callback - INFO - Starting async job submit for node 693af2e8fee8383e92b6a6bd 2025-12-11 19:41:24,981 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:41:25,024 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af2e8fee8383e92b6a6bd HTTP/1.1" 502 150 2025-12-11 19:41:25,024 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af2e8fee8383e92b6a6bd'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:41:25,024 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af2e8fee8383e92b6a6bd 2025-12-11 19:41:25,026 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af2e8fee8383e92b6a6bd HTTP/1.1" 502 150 2025-12-11 19:41:25,026 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af2e8fee8383e92b6a6bd'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:51052 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51064 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:41:26,763 - lava_callback - INFO - Received callback for node 693b18e115f407b7629df584 INFO: 172.21.0.1:51072 - "POST /node/693b18e115f407b7629df584 HTTP/1.0" 202 Accepted 2025-12-11 19:41:26,774 - lava_callback - INFO - Starting async job submit for node 693b18e115f407b7629df584 2025-12-11 19:41:26,799 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:41:26,808 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18e115f407b7629df584 HTTP/1.1" 502 150 2025-12-11 19:41:26,808 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18e115f407b7629df584'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:41:26,809 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18e115f407b7629df584 2025-12-11 19:41:26,809 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18e115f407b7629df584 HTTP/1.1" 502 150 2025-12-11 19:41:26,809 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18e115f407b7629df584'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:41:27,026 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af2e8fee8383e92b6a6bd 2025-12-11 19:41:27,027 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af2e8fee8383e92b6a6bd HTTP/1.1" 502 150 2025-12-11 19:41:27,027 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af2e8fee8383e92b6a6bd'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:41:28,809 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18e115f407b7629df584 2025-12-11 19:41:28,811 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18e115f407b7629df584 HTTP/1.1" 502 150 2025-12-11 19:41:28,811 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18e115f407b7629df584'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:41:29,328 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18e115f407b7629df580 2025-12-11 19:41:29,330 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18e115f407b7629df580 HTTP/1.1" 502 150 2025-12-11 19:41:29,330 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18e115f407b7629df580'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:41:30,437 - lava_callback - INFO - Received callback for node 693b0480fee8383e92b6f886 INFO: 172.21.0.1:35124 - "POST /node/693b0480fee8383e92b6f886 HTTP/1.0" 202 Accepted 2025-12-11 19:41:30,453 - lava_callback - INFO - Starting async job submit for node 693b0480fee8383e92b6f886 2025-12-11 19:41:30,555 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:41:30,568 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0480fee8383e92b6f886 HTTP/1.1" 502 150 2025-12-11 19:41:30,568 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0480fee8383e92b6f886'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:41:30,568 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0480fee8383e92b6f886 2025-12-11 19:41:30,569 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0480fee8383e92b6f886 HTTP/1.1" 502 150 2025-12-11 19:41:30,569 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0480fee8383e92b6f886'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:41:31,027 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af2e8fee8383e92b6a6bd 2025-12-11 19:41:31,029 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af2e8fee8383e92b6a6bd HTTP/1.1" 502 150 2025-12-11 19:41:31,029 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af2e8fee8383e92b6a6bd'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:41:32,569 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0480fee8383e92b6f886 2025-12-11 19:41:32,570 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0480fee8383e92b6f886 HTTP/1.1" 502 150 2025-12-11 19:41:32,570 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0480fee8383e92b6f886'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:41:32,811 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18e115f407b7629df584 2025-12-11 19:41:32,812 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18e115f407b7629df584 HTTP/1.1" 502 150 2025-12-11 19:41:32,812 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18e115f407b7629df584'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:35134 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:41:36,571 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0480fee8383e92b6f886 2025-12-11 19:41:36,572 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0480fee8383e92b6f886 HTTP/1.1" 502 150 2025-12-11 19:41:36,572 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0480fee8383e92b6f886'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:41:39,029 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af2e8fee8383e92b6a6bd 2025-12-11 19:41:39,035 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af2e8fee8383e92b6a6bd HTTP/1.1" 502 150 2025-12-11 19:41:39,035 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af2e8fee8383e92b6a6bd'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:41:40,813 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18e115f407b7629df584 2025-12-11 19:41:40,814 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18e115f407b7629df584 HTTP/1.1" 502 150 2025-12-11 19:41:40,814 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18e115f407b7629df584'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:41:44,572 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0480fee8383e92b6f886 2025-12-11 19:41:44,574 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0480fee8383e92b6f886 HTTP/1.1" 502 150 2025-12-11 19:41:44,574 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0480fee8383e92b6f886'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:41:45,330 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18e115f407b7629df580 2025-12-11 19:41:45,332 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18e115f407b7629df580 HTTP/1.1" 502 150 2025-12-11 19:41:45,332 - lava_callback - ERROR - Error processing callback for node 693b18e115f407b7629df580: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b18e115f407b7629df580 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b18e115f407b7629df580 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b18e115f407b7629df580 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:47982 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:41:55,035 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af2e8fee8383e92b6a6bd 2025-12-11 19:41:55,037 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af2e8fee8383e92b6a6bd HTTP/1.1" 502 150 2025-12-11 19:41:55,037 - lava_callback - ERROR - Error processing callback for node 693af2e8fee8383e92b6a6bd: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af2e8fee8383e92b6a6bd (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af2e8fee8383e92b6a6bd (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af2e8fee8383e92b6a6bd (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:36460 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:41:56,814 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18e115f407b7629df584 2025-12-11 19:41:56,815 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18e115f407b7629df584 HTTP/1.1" 502 150 2025-12-11 19:41:56,816 - lava_callback - ERROR - Error processing callback for node 693b18e115f407b7629df584: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b18e115f407b7629df584 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b18e115f407b7629df584 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b18e115f407b7629df584 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:42:00,574 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0480fee8383e92b6f886 2025-12-11 19:42:00,576 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0480fee8383e92b6f886 HTTP/1.1" 502 150 2025-12-11 19:42:00,576 - lava_callback - ERROR - Error processing callback for node 693b0480fee8383e92b6f886: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0480fee8383e92b6f886 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0480fee8383e92b6f886 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0480fee8383e92b6f886 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:42:05,104 - lava_callback - INFO - Received callback for node 693b11c6fee8383e92b739fe INFO: 172.21.0.1:52900 - "POST /node/693b11c6fee8383e92b739fe HTTP/1.0" 202 Accepted 2025-12-11 19:42:05,111 - lava_callback - INFO - Starting async job submit for node 693b11c6fee8383e92b739fe 2025-12-11 19:42:05,133 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:42:05,152 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b11c6fee8383e92b739fe HTTP/1.1" 502 150 2025-12-11 19:42:05,152 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b11c6fee8383e92b739fe'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:05,152 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b11c6fee8383e92b739fe 2025-12-11 19:42:05,152 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b11c6fee8383e92b739fe HTTP/1.1" 502 150 2025-12-11 19:42:05,152 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b11c6fee8383e92b739fe'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:52906 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:42:07,153 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b11c6fee8383e92b739fe 2025-12-11 19:42:07,154 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b11c6fee8383e92b739fe HTTP/1.1" 502 150 2025-12-11 19:42:07,154 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b11c6fee8383e92b739fe'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:10,209 - lava_callback - INFO - Received callback for node 693b0498fee8383e92b6f8d9 INFO: 172.21.0.1:38972 - "POST /node/693b0498fee8383e92b6f8d9 HTTP/1.0" 202 Accepted 2025-12-11 19:42:10,219 - lava_callback - INFO - Starting async job submit for node 693b0498fee8383e92b6f8d9 2025-12-11 19:42:10,290 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:42:10,301 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0498fee8383e92b6f8d9 HTTP/1.1" 502 150 2025-12-11 19:42:10,301 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0498fee8383e92b6f8d9'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:10,301 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0498fee8383e92b6f8d9 2025-12-11 19:42:10,302 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0498fee8383e92b6f8d9 HTTP/1.1" 502 150 2025-12-11 19:42:10,302 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0498fee8383e92b6f8d9'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:11,154 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b11c6fee8383e92b739fe 2025-12-11 19:42:11,156 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b11c6fee8383e92b739fe HTTP/1.1" 502 150 2025-12-11 19:42:11,156 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b11c6fee8383e92b739fe'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:12,302 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0498fee8383e92b6f8d9 2025-12-11 19:42:12,304 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0498fee8383e92b6f8d9 HTTP/1.1" 502 150 2025-12-11 19:42:12,304 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0498fee8383e92b6f8d9'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:38986 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 19:42:16,304 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0498fee8383e92b6f8d9 2025-12-11 19:42:16,307 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0498fee8383e92b6f8d9 HTTP/1.1" 502 150 2025-12-11 19:42:16,308 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0498fee8383e92b6f8d9'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:39000 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:42:18,789 - lava_callback - INFO - Received callback for node 693b137dfee8383e92b7417c INFO: 172.21.0.1:39004 - "POST /node/693b137dfee8383e92b7417c HTTP/1.0" 202 Accepted 2025-12-11 19:42:18,795 - lava_callback - INFO - Starting async job submit for node 693b137dfee8383e92b7417c 2025-12-11 19:42:18,816 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:42:18,834 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b137dfee8383e92b7417c HTTP/1.1" 502 150 2025-12-11 19:42:18,834 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b137dfee8383e92b7417c'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:18,834 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b137dfee8383e92b7417c 2025-12-11 19:42:18,835 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b137dfee8383e92b7417c HTTP/1.1" 502 150 2025-12-11 19:42:18,835 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b137dfee8383e92b7417c'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:19,156 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b11c6fee8383e92b739fe 2025-12-11 19:42:19,157 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b11c6fee8383e92b739fe HTTP/1.1" 502 150 2025-12-11 19:42:19,157 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b11c6fee8383e92b739fe'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:20,835 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b137dfee8383e92b7417c 2025-12-11 19:42:20,836 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b137dfee8383e92b7417c HTTP/1.1" 502 150 2025-12-11 19:42:20,836 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b137dfee8383e92b7417c'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:24,308 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0498fee8383e92b6f8d9 2025-12-11 19:42:24,309 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0498fee8383e92b6f8d9 HTTP/1.1" 502 150 2025-12-11 19:42:24,309 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0498fee8383e92b6f8d9'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:24,777 - lava_callback - INFO - Received callback for node 693afaf6fee8383e92b6d32b INFO: 172.21.0.1:51438 - "POST /node/693afaf6fee8383e92b6d32b HTTP/1.0" 202 Accepted 2025-12-11 19:42:24,789 - lava_callback - INFO - Starting async job submit for node 693afaf6fee8383e92b6d32b 2025-12-11 19:42:24,842 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b137dfee8383e92b7417c 2025-12-11 19:42:24,853 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b137dfee8383e92b7417c HTTP/1.1" 502 150 2025-12-11 19:42:24,853 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b137dfee8383e92b7417c'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:24,885 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:42:24,926 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afaf6fee8383e92b6d32b HTTP/1.1" 502 150 2025-12-11 19:42:24,927 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afaf6fee8383e92b6d32b'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:24,927 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afaf6fee8383e92b6d32b 2025-12-11 19:42:24,928 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afaf6fee8383e92b6d32b HTTP/1.1" 502 150 2025-12-11 19:42:24,928 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afaf6fee8383e92b6d32b'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:51444 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51448 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:42:26,928 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afaf6fee8383e92b6d32b 2025-12-11 19:42:26,930 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afaf6fee8383e92b6d32b HTTP/1.1" 502 150 2025-12-11 19:42:26,930 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afaf6fee8383e92b6d32b'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:29,631 - lava_callback - INFO - Received callback for node 693af6abfee8383e92b6bc97 INFO: 172.21.0.1:51464 - "POST /node/693af6abfee8383e92b6bc97 HTTP/1.0" 202 Accepted 2025-12-11 19:42:29,637 - lava_callback - INFO - Starting async job submit for node 693af6abfee8383e92b6bc97 2025-12-11 19:42:29,659 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:42:29,669 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af6abfee8383e92b6bc97 HTTP/1.1" 502 150 2025-12-11 19:42:29,669 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af6abfee8383e92b6bc97'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:29,669 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af6abfee8383e92b6bc97 2025-12-11 19:42:29,669 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af6abfee8383e92b6bc97 HTTP/1.1" 502 150 2025-12-11 19:42:29,670 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af6abfee8383e92b6bc97'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:30,930 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afaf6fee8383e92b6d32b 2025-12-11 19:42:30,931 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afaf6fee8383e92b6d32b HTTP/1.1" 502 150 2025-12-11 19:42:30,931 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afaf6fee8383e92b6d32b'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:31,670 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af6abfee8383e92b6bc97 2025-12-11 19:42:31,671 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af6abfee8383e92b6bc97 HTTP/1.1" 502 150 2025-12-11 19:42:31,671 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af6abfee8383e92b6bc97'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:32,853 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b137dfee8383e92b7417c 2025-12-11 19:42:32,854 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b137dfee8383e92b7417c HTTP/1.1" 502 150 2025-12-11 19:42:32,854 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b137dfee8383e92b7417c'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:35,157 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b11c6fee8383e92b739fe 2025-12-11 19:42:35,159 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b11c6fee8383e92b739fe HTTP/1.1" 502 150 2025-12-11 19:42:35,159 - lava_callback - ERROR - Error processing callback for node 693b11c6fee8383e92b739fe: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b11c6fee8383e92b739fe (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b11c6fee8383e92b739fe (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b11c6fee8383e92b739fe (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:42:35,671 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af6abfee8383e92b6bc97 2025-12-11 19:42:35,679 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af6abfee8383e92b6bc97 HTTP/1.1" 502 150 2025-12-11 19:42:35,679 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af6abfee8383e92b6bc97'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:50014 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:42:38,783 - lava_callback - INFO - Received callback for node 693b18e115f407b7629df582 INFO: 172.21.0.1:50030 - "POST /node/693b18e115f407b7629df582 HTTP/1.0" 202 Accepted 2025-12-11 19:42:38,792 - lava_callback - INFO - Starting async job submit for node 693b18e115f407b7629df582 2025-12-11 19:42:38,850 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:42:38,889 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18e115f407b7629df582 HTTP/1.1" 502 150 2025-12-11 19:42:38,889 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18e115f407b7629df582'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:38,889 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18e115f407b7629df582 2025-12-11 19:42:38,890 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18e115f407b7629df582 HTTP/1.1" 502 150 2025-12-11 19:42:38,890 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18e115f407b7629df582'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:38,931 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afaf6fee8383e92b6d32b 2025-12-11 19:42:38,933 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afaf6fee8383e92b6d32b HTTP/1.1" 502 150 2025-12-11 19:42:38,933 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afaf6fee8383e92b6d32b'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:40,309 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0498fee8383e92b6f8d9 2025-12-11 19:42:40,310 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0498fee8383e92b6f8d9 HTTP/1.1" 502 150 2025-12-11 19:42:40,311 - lava_callback - ERROR - Error processing callback for node 693b0498fee8383e92b6f8d9: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0498fee8383e92b6f8d9 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0498fee8383e92b6f8d9 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0498fee8383e92b6f8d9 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:42:40,890 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18e115f407b7629df582 2025-12-11 19:42:40,892 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18e115f407b7629df582 HTTP/1.1" 502 150 2025-12-11 19:42:40,892 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18e115f407b7629df582'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:43,680 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af6abfee8383e92b6bc97 2025-12-11 19:42:43,681 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af6abfee8383e92b6bc97 HTTP/1.1" 502 150 2025-12-11 19:42:43,681 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af6abfee8383e92b6bc97'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:44,892 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18e115f407b7629df582 2025-12-11 19:42:44,893 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18e115f407b7629df582 HTTP/1.1" 502 150 2025-12-11 19:42:44,893 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18e115f407b7629df582'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:44,909 - lava_callback - INFO - Received callback for node 693b190e15f407b7629df5d8 INFO: 172.21.0.1:35354 - "POST /node/693b190e15f407b7629df5d8 HTTP/1.0" 202 Accepted 2025-12-11 19:42:44,921 - lava_callback - INFO - Starting async job submit for node 693b190e15f407b7629df5d8 2025-12-11 19:42:44,937 - lava_callback - INFO - Received callback for node 693b1356fee8383e92b740db INFO: 172.21.0.1:35366 - "POST /node/693b1356fee8383e92b740db HTTP/1.0" 202 Accepted 2025-12-11 19:42:44,957 - lava_callback - INFO - Starting async job submit for node 693b1356fee8383e92b740db 2025-12-11 19:42:45,012 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:42:45,058 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b190e15f407b7629df5d8 HTTP/1.1" 502 150 2025-12-11 19:42:45,058 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b190e15f407b7629df5d8'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:45,058 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b190e15f407b7629df5d8 2025-12-11 19:42:45,067 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b190e15f407b7629df5d8 HTTP/1.1" 502 150 2025-12-11 19:42:45,067 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b190e15f407b7629df5d8'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:45,068 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:42:45,081 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1356fee8383e92b740db HTTP/1.1" 502 150 2025-12-11 19:42:45,081 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1356fee8383e92b740db'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:45,081 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1356fee8383e92b740db 2025-12-11 19:42:45,082 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1356fee8383e92b740db HTTP/1.1" 502 150 2025-12-11 19:42:45,082 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1356fee8383e92b740db'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:35380 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:42:47,067 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b190e15f407b7629df5d8 2025-12-11 19:42:47,071 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b190e15f407b7629df5d8 HTTP/1.1" 502 150 2025-12-11 19:42:47,071 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b190e15f407b7629df5d8'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:47,083 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1356fee8383e92b740db 2025-12-11 19:42:47,084 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1356fee8383e92b740db HTTP/1.1" 502 150 2025-12-11 19:42:47,084 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1356fee8383e92b740db'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:48,854 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b137dfee8383e92b7417c 2025-12-11 19:42:48,856 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b137dfee8383e92b7417c HTTP/1.1" 502 150 2025-12-11 19:42:48,856 - lava_callback - ERROR - Error processing callback for node 693b137dfee8383e92b7417c: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b137dfee8383e92b7417c (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b137dfee8383e92b7417c (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b137dfee8383e92b7417c (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:42:51,071 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b190e15f407b7629df5d8 2025-12-11 19:42:51,072 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b190e15f407b7629df5d8 HTTP/1.1" 502 150 2025-12-11 19:42:51,072 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b190e15f407b7629df5d8'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:51,084 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1356fee8383e92b740db 2025-12-11 19:42:51,085 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1356fee8383e92b740db HTTP/1.1" 502 150 2025-12-11 19:42:51,085 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1356fee8383e92b740db'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:52,893 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18e115f407b7629df582 2025-12-11 19:42:52,894 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18e115f407b7629df582 HTTP/1.1" 502 150 2025-12-11 19:42:52,895 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18e115f407b7629df582'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:54,933 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afaf6fee8383e92b6d32b 2025-12-11 19:42:54,934 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afaf6fee8383e92b6d32b HTTP/1.1" 502 150 2025-12-11 19:42:54,934 - lava_callback - ERROR - Error processing callback for node 693afaf6fee8383e92b6d32b: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afaf6fee8383e92b6d32b (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afaf6fee8383e92b6d32b (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afaf6fee8383e92b6d32b (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:36712 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:42:59,072 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b190e15f407b7629df5d8 2025-12-11 19:42:59,074 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b190e15f407b7629df5d8 HTTP/1.1" 502 150 2025-12-11 19:42:59,074 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b190e15f407b7629df5d8'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:59,086 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1356fee8383e92b740db 2025-12-11 19:42:59,086 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1356fee8383e92b740db HTTP/1.1" 502 150 2025-12-11 19:42:59,086 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1356fee8383e92b740db'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:42:59,682 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af6abfee8383e92b6bc97 2025-12-11 19:42:59,692 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af6abfee8383e92b6bc97 HTTP/1.1" 502 150 2025-12-11 19:42:59,692 - lava_callback - ERROR - Error processing callback for node 693af6abfee8383e92b6bc97: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af6abfee8383e92b6bc97 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af6abfee8383e92b6bc97 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af6abfee8383e92b6bc97 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:45034 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:43:08,895 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18e115f407b7629df582 2025-12-11 19:43:08,896 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18e115f407b7629df582 HTTP/1.1" 502 150 2025-12-11 19:43:08,896 - lava_callback - ERROR - Error processing callback for node 693b18e115f407b7629df582: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b18e115f407b7629df582 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b18e115f407b7629df582 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b18e115f407b7629df582 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:43:15,074 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b190e15f407b7629df5d8 2025-12-11 19:43:15,076 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b190e15f407b7629df5d8 HTTP/1.1" 502 150 2025-12-11 19:43:15,076 - lava_callback - ERROR - Error processing callback for node 693b190e15f407b7629df5d8: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b190e15f407b7629df5d8 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b190e15f407b7629df5d8 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b190e15f407b7629df5d8 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:43:15,087 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1356fee8383e92b740db 2025-12-11 19:43:15,088 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1356fee8383e92b740db HTTP/1.1" 502 150 2025-12-11 19:43:15,088 - lava_callback - ERROR - Error processing callback for node 693b1356fee8383e92b740db: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1356fee8383e92b740db (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1356fee8383e92b740db (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1356fee8383e92b740db (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:48200 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48204 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:43:19,917 - lava_callback - INFO - Received callback for node 693b190e15f407b7629df5d6 INFO: 172.21.0.1:38302 - "POST /node/693b190e15f407b7629df5d6 HTTP/1.0" 202 Accepted 2025-12-11 19:43:19,928 - lava_callback - INFO - Starting async job submit for node 693b190e15f407b7629df5d6 2025-12-11 19:43:20,015 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:43:20,069 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b190e15f407b7629df5d6 HTTP/1.1" 502 150 2025-12-11 19:43:20,069 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b190e15f407b7629df5d6'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:43:20,070 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b190e15f407b7629df5d6 2025-12-11 19:43:20,071 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b190e15f407b7629df5d6 HTTP/1.1" 502 150 2025-12-11 19:43:20,071 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b190e15f407b7629df5d6'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:43:22,071 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b190e15f407b7629df5d6 2025-12-11 19:43:22,072 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b190e15f407b7629df5d6 HTTP/1.1" 502 150 2025-12-11 19:43:22,072 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b190e15f407b7629df5d6'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:43:26,072 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b190e15f407b7629df5d6 2025-12-11 19:43:26,074 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b190e15f407b7629df5d6 HTTP/1.1" 502 150 2025-12-11 19:43:26,074 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b190e15f407b7629df5d6'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:38318 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38320 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:43:34,075 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b190e15f407b7629df5d6 2025-12-11 19:43:34,076 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b190e15f407b7629df5d6 HTTP/1.1" 502 150 2025-12-11 19:43:34,076 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b190e15f407b7629df5d6'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:40194 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49038 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:43:50,076 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b190e15f407b7629df5d6 2025-12-11 19:43:50,077 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b190e15f407b7629df5d6 HTTP/1.1" 502 150 2025-12-11 19:43:50,078 - lava_callback - ERROR - Error processing callback for node 693b190e15f407b7629df5d6: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b190e15f407b7629df5d6 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b190e15f407b7629df5d6 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b190e15f407b7629df5d6 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:58030 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:44:00,697 - lava_callback - INFO - Received callback for node 693af6f2fee8383e92b6bdfe INFO: 172.21.0.1:43938 - "POST /node/693af6f2fee8383e92b6bdfe HTTP/1.0" 202 Accepted 2025-12-11 19:44:00,703 - lava_callback - INFO - Starting async job submit for node 693af6f2fee8383e92b6bdfe 2025-12-11 19:44:00,724 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:44:00,737 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af6f2fee8383e92b6bdfe HTTP/1.1" 502 150 2025-12-11 19:44:00,737 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af6f2fee8383e92b6bdfe'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:44:00,737 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af6f2fee8383e92b6bdfe 2025-12-11 19:44:00,738 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af6f2fee8383e92b6bdfe HTTP/1.1" 502 150 2025-12-11 19:44:00,738 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af6f2fee8383e92b6bdfe'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:44:02,738 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af6f2fee8383e92b6bdfe 2025-12-11 19:44:02,739 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af6f2fee8383e92b6bdfe HTTP/1.1" 502 150 2025-12-11 19:44:02,739 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af6f2fee8383e92b6bdfe'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:43946 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:44:06,740 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af6f2fee8383e92b6bdfe 2025-12-11 19:44:06,741 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af6f2fee8383e92b6bdfe HTTP/1.1" 502 150 2025-12-11 19:44:06,741 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af6f2fee8383e92b6bdfe'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:44:07,402 - lava_callback - INFO - Received callback for node 693af9f4fee8383e92b6cfe3 INFO: 172.21.0.1:43950 - "POST /node/693af9f4fee8383e92b6cfe3 HTTP/1.0" 202 Accepted 2025-12-11 19:44:07,412 - lava_callback - INFO - Starting async job submit for node 693af9f4fee8383e92b6cfe3 2025-12-11 19:44:07,436 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:44:07,468 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af9f4fee8383e92b6cfe3 HTTP/1.1" 502 150 2025-12-11 19:44:07,468 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af9f4fee8383e92b6cfe3'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:44:07,468 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af9f4fee8383e92b6cfe3 2025-12-11 19:44:07,469 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af9f4fee8383e92b6cfe3 HTTP/1.1" 502 150 2025-12-11 19:44:07,469 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af9f4fee8383e92b6cfe3'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:44:09,469 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af9f4fee8383e92b6cfe3 2025-12-11 19:44:09,471 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af9f4fee8383e92b6cfe3 HTTP/1.1" 502 150 2025-12-11 19:44:09,471 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af9f4fee8383e92b6cfe3'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:44:13,471 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af9f4fee8383e92b6cfe3 2025-12-11 19:44:13,472 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af9f4fee8383e92b6cfe3 HTTP/1.1" 502 150 2025-12-11 19:44:13,472 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af9f4fee8383e92b6cfe3'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:44:14,741 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af6f2fee8383e92b6bdfe 2025-12-11 19:44:14,742 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af6f2fee8383e92b6bdfe HTTP/1.1" 502 150 2025-12-11 19:44:14,742 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af6f2fee8383e92b6bdfe'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:38280 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:38290 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:44:21,472 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af9f4fee8383e92b6cfe3 2025-12-11 19:44:21,477 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af9f4fee8383e92b6cfe3 HTTP/1.1" 502 150 2025-12-11 19:44:21,477 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af9f4fee8383e92b6cfe3'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:44:23,233 - lava_callback - INFO - Received callback for node 693afafafee8383e92b6d335 INFO: 172.21.0.1:35488 - "POST /node/693afafafee8383e92b6d335 HTTP/1.0" 202 Accepted 2025-12-11 19:44:23,241 - lava_callback - INFO - Starting async job submit for node 693afafafee8383e92b6d335 2025-12-11 19:44:23,312 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:44:23,504 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afafafee8383e92b6d335 HTTP/1.1" 502 150 2025-12-11 19:44:23,504 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afafafee8383e92b6d335'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:44:23,504 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afafafee8383e92b6d335 2025-12-11 19:44:23,505 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afafafee8383e92b6d335 HTTP/1.1" 502 150 2025-12-11 19:44:23,505 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afafafee8383e92b6d335'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:44:25,505 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afafafee8383e92b6d335 2025-12-11 19:44:25,507 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afafafee8383e92b6d335 HTTP/1.1" 502 150 2025-12-11 19:44:25,507 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afafafee8383e92b6d335'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:35492 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35498 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:44:29,507 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afafafee8383e92b6d335 2025-12-11 19:44:29,508 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afafafee8383e92b6d335 HTTP/1.1" 502 150 2025-12-11 19:44:29,509 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afafafee8383e92b6d335'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:44:30,743 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af6f2fee8383e92b6bdfe 2025-12-11 19:44:30,744 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af6f2fee8383e92b6bdfe HTTP/1.1" 502 150 2025-12-11 19:44:30,744 - lava_callback - ERROR - Error processing callback for node 693af6f2fee8383e92b6bdfe: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af6f2fee8383e92b6bdfe (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af6f2fee8383e92b6bdfe (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af6f2fee8383e92b6bdfe (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:55980 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:44:37,478 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af9f4fee8383e92b6cfe3 2025-12-11 19:44:37,479 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af9f4fee8383e92b6cfe3 HTTP/1.1" 502 150 2025-12-11 19:44:37,479 - lava_callback - ERROR - Error processing callback for node 693af9f4fee8383e92b6cfe3: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af9f4fee8383e92b6cfe3 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af9f4fee8383e92b6cfe3 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af9f4fee8383e92b6cfe3 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:44:37,509 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afafafee8383e92b6d335 2025-12-11 19:44:37,510 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afafafee8383e92b6d335 HTTP/1.1" 502 150 2025-12-11 19:44:37,510 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afafafee8383e92b6d335'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:44:39,077 - lava_callback - INFO - Received callback for node 693b190e15f407b7629df5da INFO: 172.21.0.1:55994 - "POST /node/693b190e15f407b7629df5da HTTP/1.0" 202 Accepted 2025-12-11 19:44:39,083 - lava_callback - INFO - Starting async job submit for node 693b190e15f407b7629df5da 2025-12-11 19:44:39,105 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:44:39,115 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b190e15f407b7629df5da HTTP/1.1" 502 150 2025-12-11 19:44:39,115 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b190e15f407b7629df5da'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:44:39,116 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b190e15f407b7629df5da 2025-12-11 19:44:39,116 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b190e15f407b7629df5da HTTP/1.1" 502 150 2025-12-11 19:44:39,116 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b190e15f407b7629df5da'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:44:41,116 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b190e15f407b7629df5da 2025-12-11 19:44:41,118 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b190e15f407b7629df5da HTTP/1.1" 502 150 2025-12-11 19:44:41,118 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b190e15f407b7629df5da'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:44:44,828 - lava_callback - INFO - Received callback for node 693af348fee8383e92b6a8c6 INFO: 172.21.0.1:55440 - "POST /node/693af348fee8383e92b6a8c6 HTTP/1.0" 202 Accepted 2025-12-11 19:44:44,834 - lava_callback - INFO - Starting async job submit for node 693af348fee8383e92b6a8c6 2025-12-11 19:44:44,853 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:44:44,867 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af348fee8383e92b6a8c6 HTTP/1.1" 502 150 2025-12-11 19:44:44,867 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af348fee8383e92b6a8c6'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:44:44,867 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af348fee8383e92b6a8c6 2025-12-11 19:44:44,868 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af348fee8383e92b6a8c6 HTTP/1.1" 502 150 2025-12-11 19:44:44,868 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af348fee8383e92b6a8c6'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:44:45,118 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b190e15f407b7629df5da 2025-12-11 19:44:45,119 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b190e15f407b7629df5da HTTP/1.1" 502 150 2025-12-11 19:44:45,120 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b190e15f407b7629df5da'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:55454 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:44:46,868 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af348fee8383e92b6a8c6 2025-12-11 19:44:46,869 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af348fee8383e92b6a8c6 HTTP/1.1" 502 150 2025-12-11 19:44:46,869 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af348fee8383e92b6a8c6'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:44:50,869 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af348fee8383e92b6a8c6 2025-12-11 19:44:50,871 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af348fee8383e92b6a8c6 HTTP/1.1" 502 150 2025-12-11 19:44:50,871 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af348fee8383e92b6a8c6'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:44:53,120 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b190e15f407b7629df5da 2025-12-11 19:44:53,122 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b190e15f407b7629df5da HTTP/1.1" 502 150 2025-12-11 19:44:53,122 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b190e15f407b7629df5da'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:44:53,510 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afafafee8383e92b6d335 2025-12-11 19:44:53,511 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afafafee8383e92b6d335 HTTP/1.1" 502 150 2025-12-11 19:44:53,511 - lava_callback - ERROR - Error processing callback for node 693afafafee8383e92b6d335: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afafafee8383e92b6d335 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afafafee8383e92b6d335 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afafafee8383e92b6d335 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:44:54,640 - lava_callback - INFO - Received callback for node 693b06b3fee8383e92b708b4 INFO: 172.21.0.1:60466 - "POST /node/693b06b3fee8383e92b708b4 HTTP/1.0" 202 Accepted 2025-12-11 19:44:54,650 - lava_callback - INFO - Starting async job submit for node 693b06b3fee8383e92b708b4 2025-12-11 19:44:54,707 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:44:54,720 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b06b3fee8383e92b708b4 HTTP/1.1" 502 150 2025-12-11 19:44:54,720 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b06b3fee8383e92b708b4'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:44:54,720 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b06b3fee8383e92b708b4 2025-12-11 19:44:54,721 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b06b3fee8383e92b708b4 HTTP/1.1" 502 150 2025-12-11 19:44:54,721 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b06b3fee8383e92b708b4'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:60468 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:44:56,722 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b06b3fee8383e92b708b4 2025-12-11 19:44:56,723 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b06b3fee8383e92b708b4 HTTP/1.1" 502 150 2025-12-11 19:44:56,723 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b06b3fee8383e92b708b4'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:44:58,363 - lava_callback - INFO - Received callback for node 693b191515f407b7629df5e5 INFO: 172.21.0.1:60482 - "POST /node/693b191515f407b7629df5e5 HTTP/1.0" 202 Accepted 2025-12-11 19:44:58,374 - lava_callback - INFO - Starting async job submit for node 693b191515f407b7629df5e5 2025-12-11 19:44:58,410 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:44:58,421 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b191515f407b7629df5e5 HTTP/1.1" 502 150 2025-12-11 19:44:58,422 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b191515f407b7629df5e5'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:44:58,422 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b191515f407b7629df5e5 2025-12-11 19:44:58,422 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b191515f407b7629df5e5 HTTP/1.1" 502 150 2025-12-11 19:44:58,423 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b191515f407b7629df5e5'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:44:58,871 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af348fee8383e92b6a8c6 2025-12-11 19:44:58,872 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af348fee8383e92b6a8c6 HTTP/1.1" 502 150 2025-12-11 19:44:58,873 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af348fee8383e92b6a8c6'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:00,423 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b191515f407b7629df5e5 2025-12-11 19:45:00,424 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b191515f407b7629df5e5 HTTP/1.1" 502 150 2025-12-11 19:45:00,424 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b191515f407b7629df5e5'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:00,725 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b06b3fee8383e92b708b4 2025-12-11 19:45:00,728 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b06b3fee8383e92b708b4 HTTP/1.1" 502 150 2025-12-11 19:45:00,728 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b06b3fee8383e92b708b4'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:04,424 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b191515f407b7629df5e5 2025-12-11 19:45:04,426 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b191515f407b7629df5e5 HTTP/1.1" 502 150 2025-12-11 19:45:04,426 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b191515f407b7629df5e5'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:43222 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:45:08,728 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b06b3fee8383e92b708b4 2025-12-11 19:45:08,730 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b06b3fee8383e92b708b4 HTTP/1.1" 502 150 2025-12-11 19:45:08,730 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b06b3fee8383e92b708b4'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:09,122 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b190e15f407b7629df5da 2025-12-11 19:45:09,123 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b190e15f407b7629df5da HTTP/1.1" 502 150 2025-12-11 19:45:09,124 - lava_callback - ERROR - Error processing callback for node 693b190e15f407b7629df5da: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b190e15f407b7629df5da (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b190e15f407b7629df5da (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b190e15f407b7629df5da (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:45:10,473 - lava_callback - INFO - Received callback for node 6938856b63d980a974804868 INFO: 172.21.0.1:55292 - "POST /node/6938856b63d980a974804868 HTTP/1.0" 202 Accepted 2025-12-11 19:45:10,481 - lava_callback - INFO - Starting async job submit for node 6938856b63d980a974804868 2025-12-11 19:45:10,525 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:45:10,535 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938856b63d980a974804868 HTTP/1.1" 502 150 2025-12-11 19:45:10,535 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938856b63d980a974804868'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:10,535 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938856b63d980a974804868 2025-12-11 19:45:10,536 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938856b63d980a974804868 HTTP/1.1" 502 150 2025-12-11 19:45:10,536 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938856b63d980a974804868'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:12,426 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b191515f407b7629df5e5 2025-12-11 19:45:12,428 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b191515f407b7629df5e5 HTTP/1.1" 502 150 2025-12-11 19:45:12,428 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b191515f407b7629df5e5'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:12,536 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938856b63d980a974804868 2025-12-11 19:45:12,537 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938856b63d980a974804868 HTTP/1.1" 502 150 2025-12-11 19:45:12,537 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938856b63d980a974804868'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:14,873 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af348fee8383e92b6a8c6 2025-12-11 19:45:14,874 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af348fee8383e92b6a8c6 HTTP/1.1" 502 150 2025-12-11 19:45:14,874 - lava_callback - ERROR - Error processing callback for node 693af348fee8383e92b6a8c6: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af348fee8383e92b6a8c6 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af348fee8383e92b6a8c6 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af348fee8383e92b6a8c6 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:45:14,917 - lava_callback - INFO - Received callback for node 693b191515f407b7629df5e7 INFO: 172.21.0.1:55308 - "POST /node/693b191515f407b7629df5e7 HTTP/1.0" 202 Accepted 2025-12-11 19:45:14,923 - lava_callback - INFO - Starting async job submit for node 693b191515f407b7629df5e7 2025-12-11 19:45:14,946 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:45:14,956 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b191515f407b7629df5e7 HTTP/1.1" 502 150 2025-12-11 19:45:14,956 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b191515f407b7629df5e7'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:14,956 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b191515f407b7629df5e7 2025-12-11 19:45:14,957 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b191515f407b7629df5e7 HTTP/1.1" 502 150 2025-12-11 19:45:14,957 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b191515f407b7629df5e7'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:55316 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:55326 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:45:16,537 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938856b63d980a974804868 2025-12-11 19:45:16,538 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938856b63d980a974804868 HTTP/1.1" 502 150 2025-12-11 19:45:16,538 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938856b63d980a974804868'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:16,957 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b191515f407b7629df5e7 2025-12-11 19:45:16,961 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b191515f407b7629df5e7 HTTP/1.1" 502 150 2025-12-11 19:45:16,961 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b191515f407b7629df5e7'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:17,327 - lava_callback - INFO - Received callback for node 693b137ffee8383e92b74181 INFO: 172.21.0.1:55328 - "POST /node/693b137ffee8383e92b74181 HTTP/1.0" 202 Accepted 2025-12-11 19:45:17,333 - lava_callback - INFO - Starting async job submit for node 693b137ffee8383e92b74181 2025-12-11 19:45:17,364 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:45:17,375 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b137ffee8383e92b74181 HTTP/1.1" 502 150 2025-12-11 19:45:17,375 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b137ffee8383e92b74181'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:17,376 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b137ffee8383e92b74181 2025-12-11 19:45:17,376 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b137ffee8383e92b74181 HTTP/1.1" 502 150 2025-12-11 19:45:17,376 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b137ffee8383e92b74181'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:19,377 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b137ffee8383e92b74181 2025-12-11 19:45:19,378 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b137ffee8383e92b74181 HTTP/1.1" 502 150 2025-12-11 19:45:19,378 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b137ffee8383e92b74181'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:20,963 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b191515f407b7629df5e7 2025-12-11 19:45:20,966 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b191515f407b7629df5e7 HTTP/1.1" 502 150 2025-12-11 19:45:20,966 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b191515f407b7629df5e7'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:23,378 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b137ffee8383e92b74181 2025-12-11 19:45:23,379 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b137ffee8383e92b74181 HTTP/1.1" 502 150 2025-12-11 19:45:23,379 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b137ffee8383e92b74181'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:24,538 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938856b63d980a974804868 2025-12-11 19:45:24,540 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938856b63d980a974804868 HTTP/1.1" 502 150 2025-12-11 19:45:24,540 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938856b63d980a974804868'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:24,730 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b06b3fee8383e92b708b4 2025-12-11 19:45:24,731 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b06b3fee8383e92b708b4 HTTP/1.1" 502 150 2025-12-11 19:45:24,731 - lava_callback - ERROR - Error processing callback for node 693b06b3fee8383e92b708b4: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b06b3fee8383e92b708b4 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b06b3fee8383e92b708b4 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b06b3fee8383e92b708b4 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:45:24,941 - lava_callback - INFO - Received callback for node 693ae9bffee8383e92b65e27 INFO: 172.21.0.1:44432 - "POST /node/693ae9bffee8383e92b65e27 HTTP/1.0" 202 Accepted 2025-12-11 19:45:24,990 - lava_callback - INFO - Starting async job submit for node 693ae9bffee8383e92b65e27 2025-12-11 19:45:25,015 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:45:25,043 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9bffee8383e92b65e27 HTTP/1.1" 502 150 2025-12-11 19:45:25,043 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9bffee8383e92b65e27'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:25,043 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9bffee8383e92b65e27 2025-12-11 19:45:25,044 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9bffee8383e92b65e27 HTTP/1.1" 502 150 2025-12-11 19:45:25,044 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9bffee8383e92b65e27'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:25,215 - lava_callback - INFO - Received callback for node 693ae5f1fee8383e92b64a4b INFO: 172.21.0.1:44438 - "POST /node/693ae5f1fee8383e92b64a4b HTTP/1.0" 202 Accepted 2025-12-11 19:45:25,228 - lava_callback - INFO - Starting async job submit for node 693ae5f1fee8383e92b64a4b 2025-12-11 19:45:25,319 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:45:25,363 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae5f1fee8383e92b64a4b HTTP/1.1" 502 150 2025-12-11 19:45:25,364 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae5f1fee8383e92b64a4b'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:25,364 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae5f1fee8383e92b64a4b 2025-12-11 19:45:25,365 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae5f1fee8383e92b64a4b HTTP/1.1" 502 150 2025-12-11 19:45:25,365 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae5f1fee8383e92b64a4b'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:44446 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:45:26,656 - lava_callback - INFO - Received callback for node 693b1385fee8383e92b74191 2025-12-11 19:45:26,667 - lava_callback - INFO - Starting async job submit for node 693b1385fee8383e92b74191 INFO: 172.21.0.1:44456 - "POST /node/693b1385fee8383e92b74191 HTTP/1.0" 202 Accepted 2025-12-11 19:45:26,731 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:45:26,766 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1385fee8383e92b74191 HTTP/1.1" 502 150 2025-12-11 19:45:26,766 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1385fee8383e92b74191'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:26,766 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1385fee8383e92b74191 2025-12-11 19:45:26,767 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1385fee8383e92b74191 HTTP/1.1" 502 150 2025-12-11 19:45:26,767 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1385fee8383e92b74191'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:44458 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:45:27,044 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9bffee8383e92b65e27 2025-12-11 19:45:27,046 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9bffee8383e92b65e27 HTTP/1.1" 502 150 2025-12-11 19:45:27,046 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9bffee8383e92b65e27'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:27,365 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae5f1fee8383e92b64a4b 2025-12-11 19:45:27,366 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae5f1fee8383e92b64a4b HTTP/1.1" 502 150 2025-12-11 19:45:27,366 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae5f1fee8383e92b64a4b'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:28,428 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b191515f407b7629df5e5 2025-12-11 19:45:28,429 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b191515f407b7629df5e5 HTTP/1.1" 502 150 2025-12-11 19:45:28,429 - lava_callback - ERROR - Error processing callback for node 693b191515f407b7629df5e5: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b191515f407b7629df5e5 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b191515f407b7629df5e5 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b191515f407b7629df5e5 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:45:28,767 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1385fee8383e92b74191 2025-12-11 19:45:28,768 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1385fee8383e92b74191 HTTP/1.1" 502 150 2025-12-11 19:45:28,768 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1385fee8383e92b74191'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:28,966 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b191515f407b7629df5e7 2025-12-11 19:45:28,967 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b191515f407b7629df5e7 HTTP/1.1" 502 150 2025-12-11 19:45:28,967 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b191515f407b7629df5e7'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:31,046 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9bffee8383e92b65e27 2025-12-11 19:45:31,047 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9bffee8383e92b65e27 HTTP/1.1" 502 150 2025-12-11 19:45:31,047 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9bffee8383e92b65e27'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:31,367 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae5f1fee8383e92b64a4b 2025-12-11 19:45:31,368 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae5f1fee8383e92b64a4b HTTP/1.1" 502 150 2025-12-11 19:45:31,368 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae5f1fee8383e92b64a4b'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:31,379 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b137ffee8383e92b74181 2025-12-11 19:45:31,380 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b137ffee8383e92b74181 HTTP/1.1" 502 150 2025-12-11 19:45:31,380 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b137ffee8383e92b74181'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:32,769 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1385fee8383e92b74191 2025-12-11 19:45:32,770 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1385fee8383e92b74191 HTTP/1.1" 502 150 2025-12-11 19:45:32,770 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1385fee8383e92b74191'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:52984 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:45:39,047 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9bffee8383e92b65e27 2025-12-11 19:45:39,048 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9bffee8383e92b65e27 HTTP/1.1" 502 150 2025-12-11 19:45:39,049 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9bffee8383e92b65e27'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:39,368 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae5f1fee8383e92b64a4b 2025-12-11 19:45:39,370 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae5f1fee8383e92b64a4b HTTP/1.1" 502 150 2025-12-11 19:45:39,370 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae5f1fee8383e92b64a4b'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:40,540 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938856b63d980a974804868 2025-12-11 19:45:40,541 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938856b63d980a974804868 HTTP/1.1" 502 150 2025-12-11 19:45:40,542 - lava_callback - ERROR - Error processing callback for node 6938856b63d980a974804868: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938856b63d980a974804868 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938856b63d980a974804868 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938856b63d980a974804868 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:45:40,770 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1385fee8383e92b74191 2025-12-11 19:45:40,772 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1385fee8383e92b74191 HTTP/1.1" 502 150 2025-12-11 19:45:40,772 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1385fee8383e92b74191'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:44,968 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b191515f407b7629df5e7 2025-12-11 19:45:44,969 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b191515f407b7629df5e7 HTTP/1.1" 502 150 2025-12-11 19:45:44,969 - lava_callback - ERROR - Error processing callback for node 693b191515f407b7629df5e7: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b191515f407b7629df5e7 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b191515f407b7629df5e7 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b191515f407b7629df5e7 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:45:46,331 - lava_callback - INFO - Received callback for node 693ae84cfee8383e92b65888 INFO: 172.21.0.1:56438 - "POST /node/693ae84cfee8383e92b65888 HTTP/1.0" 202 Accepted 2025-12-11 19:45:46,338 - lava_callback - INFO - Starting async job submit for node 693ae84cfee8383e92b65888 2025-12-11 19:45:46,449 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:45:46,460 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae84cfee8383e92b65888 HTTP/1.1" 502 150 2025-12-11 19:45:46,460 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae84cfee8383e92b65888'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:46,460 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae84cfee8383e92b65888 2025-12-11 19:45:46,461 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae84cfee8383e92b65888 HTTP/1.1" 502 150 2025-12-11 19:45:46,461 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae84cfee8383e92b65888'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:56450 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:45:47,381 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b137ffee8383e92b74181 2025-12-11 19:45:47,382 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b137ffee8383e92b74181 HTTP/1.1" 502 150 2025-12-11 19:45:47,382 - lava_callback - ERROR - Error processing callback for node 693b137ffee8383e92b74181: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b137ffee8383e92b74181 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b137ffee8383e92b74181 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b137ffee8383e92b74181 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:45:47,807 - lava_callback - INFO - Received callback for node 693ae330fee8383e92b640cd INFO: 172.21.0.1:56460 - "POST /node/693ae330fee8383e92b640cd HTTP/1.0" 202 Accepted 2025-12-11 19:45:47,813 - lava_callback - INFO - Starting async job submit for node 693ae330fee8383e92b640cd 2025-12-11 19:45:47,838 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:45:47,851 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae330fee8383e92b640cd HTTP/1.1" 502 150 2025-12-11 19:45:47,851 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae330fee8383e92b640cd'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:47,851 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae330fee8383e92b640cd 2025-12-11 19:45:47,852 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae330fee8383e92b640cd HTTP/1.1" 502 150 2025-12-11 19:45:47,852 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae330fee8383e92b640cd'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:48,461 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae84cfee8383e92b65888 2025-12-11 19:45:48,462 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae84cfee8383e92b65888 HTTP/1.1" 502 150 2025-12-11 19:45:48,462 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae84cfee8383e92b65888'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:49,852 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae330fee8383e92b640cd 2025-12-11 19:45:49,853 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae330fee8383e92b640cd HTTP/1.1" 502 150 2025-12-11 19:45:49,853 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae330fee8383e92b640cd'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:52,462 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae84cfee8383e92b65888 2025-12-11 19:45:52,463 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae84cfee8383e92b65888 HTTP/1.1" 502 150 2025-12-11 19:45:52,464 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae84cfee8383e92b65888'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:53,853 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae330fee8383e92b640cd 2025-12-11 19:45:53,855 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae330fee8383e92b640cd HTTP/1.1" 502 150 2025-12-11 19:45:53,855 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae330fee8383e92b640cd'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:55,049 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9bffee8383e92b65e27 2025-12-11 19:45:55,050 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9bffee8383e92b65e27 HTTP/1.1" 502 150 2025-12-11 19:45:55,050 - lava_callback - ERROR - Error processing callback for node 693ae9bffee8383e92b65e27: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9bffee8383e92b65e27 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9bffee8383e92b65e27 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9bffee8383e92b65e27 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:45:55,370 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae5f1fee8383e92b64a4b 2025-12-11 19:45:55,371 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae5f1fee8383e92b64a4b HTTP/1.1" 502 150 2025-12-11 19:45:55,371 - lava_callback - ERROR - Error processing callback for node 693ae5f1fee8383e92b64a4b: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae5f1fee8383e92b64a4b (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae5f1fee8383e92b64a4b (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae5f1fee8383e92b64a4b (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:45912 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:45:56,772 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1385fee8383e92b74191 2025-12-11 19:45:56,773 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1385fee8383e92b74191 HTTP/1.1" 502 150 2025-12-11 19:45:56,773 - lava_callback - ERROR - Error processing callback for node 693b1385fee8383e92b74191: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1385fee8383e92b74191 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1385fee8383e92b74191 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1385fee8383e92b74191 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:45:57,347 - lava_callback - INFO - Received callback for node 693b198515f407b7629df660 INFO: 172.21.0.1:45928 - "POST /node/693b198515f407b7629df660 HTTP/1.0" 202 Accepted 2025-12-11 19:45:57,357 - lava_callback - INFO - Starting async job submit for node 693b198515f407b7629df660 2025-12-11 19:45:57,405 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:45:57,422 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b198515f407b7629df660 HTTP/1.1" 502 150 2025-12-11 19:45:57,422 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b198515f407b7629df660'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:57,422 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b198515f407b7629df660 2025-12-11 19:45:57,423 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b198515f407b7629df660 HTTP/1.1" 502 150 2025-12-11 19:45:57,424 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b198515f407b7629df660'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:45:59,424 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b198515f407b7629df660 2025-12-11 19:45:59,425 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b198515f407b7629df660 HTTP/1.1" 502 150 2025-12-11 19:45:59,425 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b198515f407b7629df660'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:46:00,464 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae84cfee8383e92b65888 2025-12-11 19:46:00,465 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae84cfee8383e92b65888 HTTP/1.1" 502 150 2025-12-11 19:46:00,465 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae84cfee8383e92b65888'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:46:01,855 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae330fee8383e92b640cd 2025-12-11 19:46:01,856 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae330fee8383e92b640cd HTTP/1.1" 502 150 2025-12-11 19:46:01,856 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae330fee8383e92b640cd'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:46:03,425 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b198515f407b7629df660 2025-12-11 19:46:03,427 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b198515f407b7629df660 HTTP/1.1" 502 150 2025-12-11 19:46:03,427 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b198515f407b7629df660'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:46:05,832 - lava_callback - INFO - Received callback for node 693b08f3fee8383e92b71483 INFO: 172.21.0.1:54960 - "POST /node/693b08f3fee8383e92b71483 HTTP/1.0" 202 Accepted 2025-12-11 19:46:05,844 - lava_callback - INFO - Starting async job submit for node 693b08f3fee8383e92b71483 2025-12-11 19:46:05,906 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:46:05,918 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b08f3fee8383e92b71483 HTTP/1.1" 502 150 2025-12-11 19:46:05,918 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b08f3fee8383e92b71483'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:46:05,919 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b08f3fee8383e92b71483 2025-12-11 19:46:05,919 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b08f3fee8383e92b71483 HTTP/1.1" 502 150 2025-12-11 19:46:05,919 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b08f3fee8383e92b71483'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:54962 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:46:07,920 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b08f3fee8383e92b71483 2025-12-11 19:46:07,921 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b08f3fee8383e92b71483 HTTP/1.1" 502 150 2025-12-11 19:46:07,921 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b08f3fee8383e92b71483'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:46:11,427 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b198515f407b7629df660 2025-12-11 19:46:11,428 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b198515f407b7629df660 HTTP/1.1" 502 150 2025-12-11 19:46:11,428 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b198515f407b7629df660'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:46:11,921 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b08f3fee8383e92b71483 2025-12-11 19:46:11,922 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b08f3fee8383e92b71483 HTTP/1.1" 502 150 2025-12-11 19:46:11,922 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b08f3fee8383e92b71483'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:55590 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 19:46:16,465 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae84cfee8383e92b65888 2025-12-11 19:46:16,467 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae84cfee8383e92b65888 HTTP/1.1" 502 150 2025-12-11 19:46:16,467 - lava_callback - ERROR - Error processing callback for node 693ae84cfee8383e92b65888: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae84cfee8383e92b65888 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae84cfee8383e92b65888 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae84cfee8383e92b65888 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:55604 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:46:17,856 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae330fee8383e92b640cd 2025-12-11 19:46:17,858 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae330fee8383e92b640cd HTTP/1.1" 502 150 2025-12-11 19:46:17,858 - lava_callback - ERROR - Error processing callback for node 693ae330fee8383e92b640cd: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae330fee8383e92b640cd (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae330fee8383e92b640cd (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae330fee8383e92b640cd (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:46:19,923 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b08f3fee8383e92b71483 2025-12-11 19:46:19,924 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b08f3fee8383e92b71483 HTTP/1.1" 502 150 2025-12-11 19:46:19,924 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b08f3fee8383e92b71483'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:55124 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55130 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:46:27,429 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b198515f407b7629df660 2025-12-11 19:46:27,430 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b198515f407b7629df660 HTTP/1.1" 502 150 2025-12-11 19:46:27,430 - lava_callback - ERROR - Error processing callback for node 693b198515f407b7629df660: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b198515f407b7629df660 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b198515f407b7629df660 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b198515f407b7629df660 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:46:32,859 - lava_callback - INFO - Received callback for node 693b191515f407b7629df5e9 INFO: 172.21.0.1:39920 - "POST /node/693b191515f407b7629df5e9 HTTP/1.0" 202 Accepted 2025-12-11 19:46:32,866 - lava_callback - INFO - Starting async job submit for node 693b191515f407b7629df5e9 2025-12-11 19:46:32,889 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:46:32,912 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b191515f407b7629df5e9 HTTP/1.1" 502 150 2025-12-11 19:46:32,912 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b191515f407b7629df5e9'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:46:32,912 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b191515f407b7629df5e9 2025-12-11 19:46:32,913 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b191515f407b7629df5e9 HTTP/1.1" 502 150 2025-12-11 19:46:32,913 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b191515f407b7629df5e9'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:46:34,913 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b191515f407b7629df5e9 2025-12-11 19:46:34,914 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b191515f407b7629df5e9 HTTP/1.1" 502 150 2025-12-11 19:46:34,914 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b191515f407b7629df5e9'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:46:35,924 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b08f3fee8383e92b71483 2025-12-11 19:46:35,925 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b08f3fee8383e92b71483 HTTP/1.1" 502 150 2025-12-11 19:46:35,925 - lava_callback - ERROR - Error processing callback for node 693b08f3fee8383e92b71483: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b08f3fee8383e92b71483 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b08f3fee8383e92b71483 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b08f3fee8383e92b71483 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:39930 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:46:38,914 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b191515f407b7629df5e9 2025-12-11 19:46:38,916 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b191515f407b7629df5e9 HTTP/1.1" 502 150 2025-12-11 19:46:38,916 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b191515f407b7629df5e9'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:46:45,939 - lava_callback - INFO - Received callback for node 693ae84efee8383e92b6588f INFO: 172.21.0.1:34934 - "POST /node/693ae84efee8383e92b6588f HTTP/1.0" 202 Accepted 2025-12-11 19:46:45,950 - lava_callback - INFO - Starting async job submit for node 693ae84efee8383e92b6588f 2025-12-11 19:46:46,063 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:46:46,073 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae84efee8383e92b6588f HTTP/1.1" 502 150 2025-12-11 19:46:46,073 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae84efee8383e92b6588f'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:46:46,074 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae84efee8383e92b6588f 2025-12-11 19:46:46,074 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae84efee8383e92b6588f HTTP/1.1" 502 150 2025-12-11 19:46:46,074 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae84efee8383e92b6588f'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:34946 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:46:46,916 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b191515f407b7629df5e9 2025-12-11 19:46:46,917 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b191515f407b7629df5e9 HTTP/1.1" 502 150 2025-12-11 19:46:46,917 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b191515f407b7629df5e9'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:46:48,075 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae84efee8383e92b6588f 2025-12-11 19:46:48,076 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae84efee8383e92b6588f HTTP/1.1" 502 150 2025-12-11 19:46:48,076 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae84efee8383e92b6588f'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:46:52,076 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae84efee8383e92b6588f 2025-12-11 19:46:52,077 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae84efee8383e92b6588f HTTP/1.1" 502 150 2025-12-11 19:46:52,077 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae84efee8383e92b6588f'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:37902 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:46:57,628 - lava_callback - INFO - Received callback for node 693afbb9fee8383e92b6d5a9 INFO: 172.21.0.1:37910 - "POST /node/693afbb9fee8383e92b6d5a9 HTTP/1.0" 202 Accepted 2025-12-11 19:46:57,655 - lava_callback - INFO - Starting async job submit for node 693afbb9fee8383e92b6d5a9 2025-12-11 19:46:57,709 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:46:57,738 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afbb9fee8383e92b6d5a9 HTTP/1.1" 502 150 2025-12-11 19:46:57,738 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afbb9fee8383e92b6d5a9'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:46:57,738 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afbb9fee8383e92b6d5a9 2025-12-11 19:46:57,739 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afbb9fee8383e92b6d5a9 HTTP/1.1" 502 150 2025-12-11 19:46:57,739 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afbb9fee8383e92b6d5a9'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:46:59,739 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afbb9fee8383e92b6d5a9 2025-12-11 19:46:59,740 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afbb9fee8383e92b6d5a9 HTTP/1.1" 502 150 2025-12-11 19:46:59,741 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afbb9fee8383e92b6d5a9'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:00,078 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae84efee8383e92b6588f 2025-12-11 19:47:00,079 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae84efee8383e92b6588f HTTP/1.1" 502 150 2025-12-11 19:47:00,079 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae84efee8383e92b6588f'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:02,917 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b191515f407b7629df5e9 2025-12-11 19:47:02,918 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b191515f407b7629df5e9 HTTP/1.1" 502 150 2025-12-11 19:47:02,918 - lava_callback - ERROR - Error processing callback for node 693b191515f407b7629df5e9: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b191515f407b7629df5e9 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b191515f407b7629df5e9 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b191515f407b7629df5e9 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:47:03,741 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afbb9fee8383e92b6d5a9 2025-12-11 19:47:03,746 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afbb9fee8383e92b6d5a9 HTTP/1.1" 502 150 2025-12-11 19:47:03,746 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afbb9fee8383e92b6d5a9'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:05,294 - lava_callback - INFO - Received callback for node 693ae843fee8383e92b65862 INFO: 172.21.0.1:39172 - "POST /node/693ae843fee8383e92b65862 HTTP/1.0" 202 Accepted 2025-12-11 19:47:05,302 - lava_callback - INFO - Starting async job submit for node 693ae843fee8383e92b65862 2025-12-11 19:47:05,417 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:47:05,428 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae843fee8383e92b65862 HTTP/1.1" 502 150 2025-12-11 19:47:05,428 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae843fee8383e92b65862'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:05,428 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae843fee8383e92b65862 2025-12-11 19:47:05,429 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae843fee8383e92b65862 HTTP/1.1" 502 150 2025-12-11 19:47:05,429 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae843fee8383e92b65862'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:39174 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:47:07,429 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae843fee8383e92b65862 2025-12-11 19:47:07,436 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae843fee8383e92b65862 HTTP/1.1" 502 150 2025-12-11 19:47:07,436 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae843fee8383e92b65862'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:11,437 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae843fee8383e92b65862 2025-12-11 19:47:11,438 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae843fee8383e92b65862 HTTP/1.1" 502 150 2025-12-11 19:47:11,438 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae843fee8383e92b65862'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:11,746 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afbb9fee8383e92b6d5a9 2025-12-11 19:47:11,747 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afbb9fee8383e92b6d5a9 HTTP/1.1" 502 150 2025-12-11 19:47:11,748 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afbb9fee8383e92b6d5a9'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:11,831 - lava_callback - INFO - Received callback for node 693afb31fee8383e92b6d3d0 INFO: 172.21.0.1:37218 - "POST /node/693afb31fee8383e92b6d3d0 HTTP/1.0" 202 Accepted 2025-12-11 19:47:11,845 - lava_callback - INFO - Starting async job submit for node 693afb31fee8383e92b6d3d0 2025-12-11 19:47:11,878 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:47:11,891 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afb31fee8383e92b6d3d0 HTTP/1.1" 502 150 2025-12-11 19:47:11,892 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afb31fee8383e92b6d3d0'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:11,892 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afb31fee8383e92b6d3d0 2025-12-11 19:47:11,892 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afb31fee8383e92b6d3d0 HTTP/1.1" 502 150 2025-12-11 19:47:11,893 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afb31fee8383e92b6d3d0'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:13,893 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afb31fee8383e92b6d3d0 2025-12-11 19:47:13,894 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afb31fee8383e92b6d3d0 HTTP/1.1" 502 150 2025-12-11 19:47:13,894 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afb31fee8383e92b6d3d0'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:15,083 - lava_callback - INFO - Received callback for node 693b198515f407b7629df662 INFO: 172.21.0.1:37228 - "POST /node/693b198515f407b7629df662 HTTP/1.0" 202 Accepted 2025-12-11 19:47:15,090 - lava_callback - INFO - Starting async job submit for node 693b198515f407b7629df662 2025-12-11 19:47:15,112 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:47:15,121 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b198515f407b7629df662 HTTP/1.1" 502 150 2025-12-11 19:47:15,121 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b198515f407b7629df662'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:15,121 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b198515f407b7629df662 2025-12-11 19:47:15,122 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b198515f407b7629df662 HTTP/1.1" 502 150 2025-12-11 19:47:15,122 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b198515f407b7629df662'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:37240 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 19:47:16,079 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae84efee8383e92b6588f 2025-12-11 19:47:16,081 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae84efee8383e92b6588f HTTP/1.1" 502 150 2025-12-11 19:47:16,081 - lava_callback - ERROR - Error processing callback for node 693ae84efee8383e92b6588f: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae84efee8383e92b6588f (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae84efee8383e92b6588f (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae84efee8383e92b6588f (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:37242 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:47:17,122 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b198515f407b7629df662 2025-12-11 19:47:17,124 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b198515f407b7629df662 HTTP/1.1" 502 150 2025-12-11 19:47:17,124 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b198515f407b7629df662'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:17,894 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afb31fee8383e92b6d3d0 2025-12-11 19:47:17,895 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afb31fee8383e92b6d3d0 HTTP/1.1" 502 150 2025-12-11 19:47:17,895 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afb31fee8383e92b6d3d0'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:17,999 - lava_callback - INFO - Received callback for node 693b13fefee8383e92b742b9 INFO: 172.21.0.1:37246 - "POST /node/693b13fefee8383e92b742b9 HTTP/1.0" 202 Accepted 2025-12-11 19:47:18,006 - lava_callback - INFO - Starting async job submit for node 693b13fefee8383e92b742b9 2025-12-11 19:47:18,028 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:47:18,065 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b13fefee8383e92b742b9 HTTP/1.1" 502 150 2025-12-11 19:47:18,065 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b13fefee8383e92b742b9'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:18,066 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b13fefee8383e92b742b9 2025-12-11 19:47:18,066 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b13fefee8383e92b742b9 HTTP/1.1" 502 150 2025-12-11 19:47:18,066 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b13fefee8383e92b742b9'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:19,438 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae843fee8383e92b65862 2025-12-11 19:47:19,439 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae843fee8383e92b65862 HTTP/1.1" 502 150 2025-12-11 19:47:19,440 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae843fee8383e92b65862'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:20,067 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b13fefee8383e92b742b9 2025-12-11 19:47:20,068 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b13fefee8383e92b742b9 HTTP/1.1" 502 150 2025-12-11 19:47:20,068 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b13fefee8383e92b742b9'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:21,124 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b198515f407b7629df662 2025-12-11 19:47:21,125 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b198515f407b7629df662 HTTP/1.1" 502 150 2025-12-11 19:47:21,125 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b198515f407b7629df662'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:24,068 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b13fefee8383e92b742b9 2025-12-11 19:47:24,070 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b13fefee8383e92b742b9 HTTP/1.1" 502 150 2025-12-11 19:47:24,070 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b13fefee8383e92b742b9'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:24,905 - lava_callback - INFO - Received callback for node 693b13defee8383e92b74293 INFO: 172.21.0.1:43404 - "POST /node/693b13defee8383e92b74293 HTTP/1.0" 202 Accepted 2025-12-11 19:47:24,911 - lava_callback - INFO - Starting async job submit for node 693b13defee8383e92b74293 2025-12-11 19:47:24,933 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:47:24,951 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b13defee8383e92b74293 HTTP/1.1" 502 150 2025-12-11 19:47:24,951 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b13defee8383e92b74293'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:24,951 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b13defee8383e92b74293 2025-12-11 19:47:24,952 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b13defee8383e92b74293 HTTP/1.1" 502 150 2025-12-11 19:47:24,952 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b13defee8383e92b74293'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:25,895 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afb31fee8383e92b6d3d0 2025-12-11 19:47:25,896 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afb31fee8383e92b6d3d0 HTTP/1.1" 502 150 2025-12-11 19:47:25,897 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afb31fee8383e92b6d3d0'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:43420 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43422 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:47:26,952 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b13defee8383e92b74293 2025-12-11 19:47:26,953 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b13defee8383e92b74293 HTTP/1.1" 502 150 2025-12-11 19:47:26,954 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b13defee8383e92b74293'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:26,961 - lava_callback - INFO - Received callback for node 693ae331fee8383e92b640d0 INFO: 172.21.0.1:43436 - "POST /node/693ae331fee8383e92b640d0 HTTP/1.0" 202 Accepted 2025-12-11 19:47:26,972 - lava_callback - INFO - Starting async job submit for node 693ae331fee8383e92b640d0 2025-12-11 19:47:27,005 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:47:27,025 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae331fee8383e92b640d0 HTTP/1.1" 502 150 2025-12-11 19:47:27,025 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae331fee8383e92b640d0'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:27,026 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae331fee8383e92b640d0 2025-12-11 19:47:27,026 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae331fee8383e92b640d0 HTTP/1.1" 502 150 2025-12-11 19:47:27,026 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae331fee8383e92b640d0'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:27,748 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afbb9fee8383e92b6d5a9 2025-12-11 19:47:27,749 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afbb9fee8383e92b6d5a9 HTTP/1.1" 502 150 2025-12-11 19:47:27,749 - lava_callback - ERROR - Error processing callback for node 693afbb9fee8383e92b6d5a9: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afbb9fee8383e92b6d5a9 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afbb9fee8383e92b6d5a9 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afbb9fee8383e92b6d5a9 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:47:29,027 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae331fee8383e92b640d0 2025-12-11 19:47:29,028 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae331fee8383e92b640d0 HTTP/1.1" 502 150 2025-12-11 19:47:29,028 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae331fee8383e92b640d0'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:29,126 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b198515f407b7629df662 2025-12-11 19:47:29,126 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b198515f407b7629df662 HTTP/1.1" 502 150 2025-12-11 19:47:29,127 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b198515f407b7629df662'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:30,954 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b13defee8383e92b74293 2025-12-11 19:47:30,955 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b13defee8383e92b74293 HTTP/1.1" 502 150 2025-12-11 19:47:30,955 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b13defee8383e92b74293'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:32,070 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b13fefee8383e92b742b9 2025-12-11 19:47:32,071 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b13fefee8383e92b742b9 HTTP/1.1" 502 150 2025-12-11 19:47:32,071 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b13fefee8383e92b742b9'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:33,028 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae331fee8383e92b640d0 2025-12-11 19:47:33,029 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae331fee8383e92b640d0 HTTP/1.1" 502 150 2025-12-11 19:47:33,029 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae331fee8383e92b640d0'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:35,440 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae843fee8383e92b65862 2025-12-11 19:47:35,441 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae843fee8383e92b65862 HTTP/1.1" 502 150 2025-12-11 19:47:35,441 - lava_callback - ERROR - Error processing callback for node 693ae843fee8383e92b65862: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae843fee8383e92b65862 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae843fee8383e92b65862 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae843fee8383e92b65862 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:56148 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:47:38,955 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b13defee8383e92b74293 2025-12-11 19:47:38,957 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b13defee8383e92b74293 HTTP/1.1" 502 150 2025-12-11 19:47:38,957 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b13defee8383e92b74293'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:41,030 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae331fee8383e92b640d0 2025-12-11 19:47:41,031 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae331fee8383e92b640d0 HTTP/1.1" 502 150 2025-12-11 19:47:41,031 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae331fee8383e92b640d0'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:41,897 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afb31fee8383e92b6d3d0 2025-12-11 19:47:41,898 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afb31fee8383e92b6d3d0 HTTP/1.1" 502 150 2025-12-11 19:47:41,898 - lava_callback - ERROR - Error processing callback for node 693afb31fee8383e92b6d3d0: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afb31fee8383e92b6d3d0 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afb31fee8383e92b6d3d0 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afb31fee8383e92b6d3d0 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:47:45,127 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b198515f407b7629df662 2025-12-11 19:47:45,128 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b198515f407b7629df662 HTTP/1.1" 502 150 2025-12-11 19:47:45,128 - lava_callback - ERROR - Error processing callback for node 693b198515f407b7629df662: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b198515f407b7629df662 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b198515f407b7629df662 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b198515f407b7629df662 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:47:45,299 - lava_callback - INFO - Received callback for node 693ae641fee8383e92b64c71 INFO: 172.21.0.1:35182 - "POST /node/693ae641fee8383e92b64c71 HTTP/1.0" 202 Accepted 2025-12-11 19:47:45,309 - lava_callback - INFO - Starting async job submit for node 693ae641fee8383e92b64c71 2025-12-11 19:47:45,349 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:47:45,370 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae641fee8383e92b64c71 HTTP/1.1" 502 150 2025-12-11 19:47:45,370 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae641fee8383e92b64c71'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:45,370 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae641fee8383e92b64c71 2025-12-11 19:47:45,371 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae641fee8383e92b64c71 HTTP/1.1" 502 150 2025-12-11 19:47:45,371 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae641fee8383e92b64c71'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:35194 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:47:47,371 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae641fee8383e92b64c71 2025-12-11 19:47:47,372 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae641fee8383e92b64c71 HTTP/1.1" 502 150 2025-12-11 19:47:47,373 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae641fee8383e92b64c71'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:48,072 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b13fefee8383e92b742b9 2025-12-11 19:47:48,073 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b13fefee8383e92b742b9 HTTP/1.1" 502 150 2025-12-11 19:47:48,073 - lava_callback - ERROR - Error processing callback for node 693b13fefee8383e92b742b9: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b13fefee8383e92b742b9 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b13fefee8383e92b742b9 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b13fefee8383e92b742b9 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:47:51,373 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae641fee8383e92b64c71 2025-12-11 19:47:51,374 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae641fee8383e92b64c71 HTTP/1.1" 502 150 2025-12-11 19:47:51,374 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae641fee8383e92b64c71'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:47:54,957 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b13defee8383e92b74293 2025-12-11 19:47:54,958 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b13defee8383e92b74293 HTTP/1.1" 502 150 2025-12-11 19:47:54,958 - lava_callback - ERROR - Error processing callback for node 693b13defee8383e92b74293: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b13defee8383e92b74293 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b13defee8383e92b74293 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b13defee8383e92b74293 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:45536 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:47:57,031 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae331fee8383e92b640d0 2025-12-11 19:47:57,033 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae331fee8383e92b640d0 HTTP/1.1" 502 150 2025-12-11 19:47:57,033 - lava_callback - ERROR - Error processing callback for node 693ae331fee8383e92b640d0: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae331fee8383e92b640d0 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae331fee8383e92b640d0 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae331fee8383e92b640d0 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:47:59,374 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae641fee8383e92b64c71 2025-12-11 19:47:59,375 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae641fee8383e92b64c71 HTTP/1.1" 502 150 2025-12-11 19:47:59,376 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae641fee8383e92b64c71'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:48:01,156 - lava_callback - INFO - Received callback for node 693b198515f407b7629df661 INFO: 172.21.0.1:46424 - "POST /node/693b198515f407b7629df661 HTTP/1.0" 202 Accepted 2025-12-11 19:48:01,163 - lava_callback - INFO - Starting async job submit for node 693b198515f407b7629df661 2025-12-11 19:48:01,184 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:48:01,194 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b198515f407b7629df661 HTTP/1.1" 502 150 2025-12-11 19:48:01,194 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b198515f407b7629df661'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:48:01,194 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b198515f407b7629df661 2025-12-11 19:48:01,195 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b198515f407b7629df661 HTTP/1.1" 502 150 2025-12-11 19:48:01,195 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b198515f407b7629df661'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:48:03,195 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b198515f407b7629df661 2025-12-11 19:48:03,196 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b198515f407b7629df661 HTTP/1.1" 502 150 2025-12-11 19:48:03,196 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b198515f407b7629df661'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:48:04,991 - lava_callback - INFO - Received callback for node 693af36dfee8383e92b6a93e INFO: 172.21.0.1:46426 - "POST /node/693af36dfee8383e92b6a93e HTTP/1.0" 202 Accepted 2025-12-11 19:48:05,001 - lava_callback - INFO - Starting async job submit for node 693af36dfee8383e92b6a93e 2025-12-11 19:48:05,046 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:48:05,088 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af36dfee8383e92b6a93e HTTP/1.1" 502 150 2025-12-11 19:48:05,088 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af36dfee8383e92b6a93e'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:48:05,088 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af36dfee8383e92b6a93e 2025-12-11 19:48:05,089 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af36dfee8383e92b6a93e HTTP/1.1" 502 150 2025-12-11 19:48:05,089 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af36dfee8383e92b6a93e'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:46434 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:48:07,090 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af36dfee8383e92b6a93e 2025-12-11 19:48:07,091 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af36dfee8383e92b6a93e HTTP/1.1" 502 150 2025-12-11 19:48:07,091 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af36dfee8383e92b6a93e'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:48:07,197 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b198515f407b7629df661 2025-12-11 19:48:07,198 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b198515f407b7629df661 HTTP/1.1" 502 150 2025-12-11 19:48:07,198 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b198515f407b7629df661'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:48:11,091 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af36dfee8383e92b6a93e 2025-12-11 19:48:11,093 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af36dfee8383e92b6a93e HTTP/1.1" 502 150 2025-12-11 19:48:11,094 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af36dfee8383e92b6a93e'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:37662 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 19:48:15,199 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b198515f407b7629df661 2025-12-11 19:48:15,200 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b198515f407b7629df661 HTTP/1.1" 502 150 2025-12-11 19:48:15,200 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b198515f407b7629df661'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:48:15,376 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae641fee8383e92b64c71 2025-12-11 19:48:15,378 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae641fee8383e92b64c71 HTTP/1.1" 502 150 2025-12-11 19:48:15,378 - lava_callback - ERROR - Error processing callback for node 693ae641fee8383e92b64c71: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae641fee8383e92b64c71 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae641fee8383e92b64c71 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae641fee8383e92b64c71 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:37676 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:48:19,094 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af36dfee8383e92b6a93e 2025-12-11 19:48:19,095 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af36dfee8383e92b6a93e HTTP/1.1" 502 150 2025-12-11 19:48:19,095 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af36dfee8383e92b6a93e'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:48:25,544 - lava_callback - INFO - Received callback for node 693b13aefee8383e92b741db INFO: 172.21.0.1:44622 - "POST /node/693b13aefee8383e92b741db HTTP/1.0" 202 Accepted 2025-12-11 19:48:25,550 - lava_callback - INFO - Starting async job submit for node 693b13aefee8383e92b741db 2025-12-11 19:48:25,591 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:48:25,611 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b13aefee8383e92b741db HTTP/1.1" 502 150 2025-12-11 19:48:25,611 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b13aefee8383e92b741db'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:48:25,612 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b13aefee8383e92b741db 2025-12-11 19:48:25,612 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b13aefee8383e92b741db HTTP/1.1" 502 150 2025-12-11 19:48:25,612 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b13aefee8383e92b741db'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:44638 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44648 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:48:27,613 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b13aefee8383e92b741db 2025-12-11 19:48:27,614 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b13aefee8383e92b741db HTTP/1.1" 502 150 2025-12-11 19:48:27,614 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b13aefee8383e92b741db'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:48:31,200 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b198515f407b7629df661 2025-12-11 19:48:31,202 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b198515f407b7629df661 HTTP/1.1" 502 150 2025-12-11 19:48:31,202 - lava_callback - ERROR - Error processing callback for node 693b198515f407b7629df661: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b198515f407b7629df661 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b198515f407b7629df661 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b198515f407b7629df661 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:48:31,614 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b13aefee8383e92b741db 2025-12-11 19:48:31,615 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b13aefee8383e92b741db HTTP/1.1" 502 150 2025-12-11 19:48:31,615 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b13aefee8383e92b741db'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:48:35,095 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af36dfee8383e92b6a93e 2025-12-11 19:48:35,097 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af36dfee8383e92b6a93e HTTP/1.1" 502 150 2025-12-11 19:48:35,097 - lava_callback - ERROR - Error processing callback for node 693af36dfee8383e92b6a93e: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af36dfee8383e92b6a93e (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af36dfee8383e92b6a93e (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af36dfee8383e92b6a93e (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:48:35,604 - lava_callback - INFO - Received callback for node 693b199115f407b7629df680 INFO: 172.21.0.1:39946 - "POST /node/693b199115f407b7629df680 HTTP/1.0" 202 Accepted 2025-12-11 19:48:35,611 - lava_callback - INFO - Starting async job submit for node 693b199115f407b7629df680 2025-12-11 19:48:35,633 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:48:35,646 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199115f407b7629df680 HTTP/1.1" 502 150 2025-12-11 19:48:35,647 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199115f407b7629df680'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:48:35,647 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199115f407b7629df680 2025-12-11 19:48:35,647 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199115f407b7629df680 HTTP/1.1" 502 150 2025-12-11 19:48:35,648 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199115f407b7629df680'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:39950 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:48:37,650 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199115f407b7629df680 2025-12-11 19:48:37,653 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199115f407b7629df680 HTTP/1.1" 502 150 2025-12-11 19:48:37,653 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199115f407b7629df680'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:48:39,616 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b13aefee8383e92b741db 2025-12-11 19:48:39,617 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b13aefee8383e92b741db HTTP/1.1" 502 150 2025-12-11 19:48:39,617 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b13aefee8383e92b741db'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:48:41,654 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199115f407b7629df680 2025-12-11 19:48:41,655 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199115f407b7629df680 HTTP/1.1" 502 150 2025-12-11 19:48:41,655 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199115f407b7629df680'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:36764 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:48:49,656 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199115f407b7629df680 2025-12-11 19:48:49,657 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199115f407b7629df680 HTTP/1.1" 502 150 2025-12-11 19:48:49,657 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199115f407b7629df680'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:48:53,933 - lava_callback - INFO - Received callback for node 693ae331fee8383e92b640d3 INFO: 172.21.0.1:43954 - "POST /node/693ae331fee8383e92b640d3 HTTP/1.0" 202 Accepted 2025-12-11 19:48:53,939 - lava_callback - INFO - Starting async job submit for node 693ae331fee8383e92b640d3 2025-12-11 19:48:53,964 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:48:53,975 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae331fee8383e92b640d3 HTTP/1.1" 502 150 2025-12-11 19:48:53,975 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae331fee8383e92b640d3'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:48:53,975 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae331fee8383e92b640d3 2025-12-11 19:48:53,976 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae331fee8383e92b640d3 HTTP/1.1" 502 150 2025-12-11 19:48:53,976 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae331fee8383e92b640d3'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:48:55,617 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b13aefee8383e92b741db 2025-12-11 19:48:55,618 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b13aefee8383e92b741db HTTP/1.1" 502 150 2025-12-11 19:48:55,618 - lava_callback - ERROR - Error processing callback for node 693b13aefee8383e92b741db: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b13aefee8383e92b741db (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b13aefee8383e92b741db (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b13aefee8383e92b741db (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:48:55,976 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae331fee8383e92b640d3 2025-12-11 19:48:55,977 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae331fee8383e92b640d3 HTTP/1.1" 502 150 2025-12-11 19:48:55,978 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae331fee8383e92b640d3'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:43956 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:48:59,978 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae331fee8383e92b640d3 2025-12-11 19:48:59,979 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae331fee8383e92b640d3 HTTP/1.1" 502 150 2025-12-11 19:48:59,979 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae331fee8383e92b640d3'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:49:05,470 - lava_callback - INFO - Received callback for node 693b140afee8383e92b742dd INFO: 172.21.0.1:56756 - "POST /node/693b140afee8383e92b742dd HTTP/1.0" 202 Accepted 2025-12-11 19:49:05,476 - lava_callback - INFO - Starting async job submit for node 693b140afee8383e92b742dd 2025-12-11 19:49:05,498 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:49:05,511 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b140afee8383e92b742dd HTTP/1.1" 502 150 2025-12-11 19:49:05,511 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b140afee8383e92b742dd'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:49:05,511 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b140afee8383e92b742dd 2025-12-11 19:49:05,511 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b140afee8383e92b742dd HTTP/1.1" 502 150 2025-12-11 19:49:05,511 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b140afee8383e92b742dd'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:49:05,657 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199115f407b7629df680 2025-12-11 19:49:05,658 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199115f407b7629df680 HTTP/1.1" 502 150 2025-12-11 19:49:05,659 - lava_callback - ERROR - Error processing callback for node 693b199115f407b7629df680: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b199115f407b7629df680 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b199115f407b7629df680 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b199115f407b7629df680 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:56760 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:49:07,512 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b140afee8383e92b742dd 2025-12-11 19:49:07,513 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b140afee8383e92b742dd HTTP/1.1" 502 150 2025-12-11 19:49:07,513 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b140afee8383e92b742dd'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:49:07,979 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae331fee8383e92b640d3 2025-12-11 19:49:07,983 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae331fee8383e92b640d3 HTTP/1.1" 502 150 2025-12-11 19:49:07,983 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae331fee8383e92b640d3'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:49:11,513 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b140afee8383e92b742dd 2025-12-11 19:49:11,514 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b140afee8383e92b742dd HTTP/1.1" 502 150 2025-12-11 19:49:11,514 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b140afee8383e92b742dd'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:50718 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:50720 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:49:17,006 - lava_callback - INFO - Received callback for node 693b199215f407b7629df682 INFO: 172.21.0.1:50734 - "POST /node/693b199215f407b7629df682 HTTP/1.0" 202 Accepted 2025-12-11 19:49:17,013 - lava_callback - INFO - Starting async job submit for node 693b199215f407b7629df682 2025-12-11 19:49:17,038 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:49:17,075 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199215f407b7629df682 HTTP/1.1" 502 150 2025-12-11 19:49:17,075 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199215f407b7629df682'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:49:17,076 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199215f407b7629df682 2025-12-11 19:49:17,076 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199215f407b7629df682 HTTP/1.1" 502 150 2025-12-11 19:49:17,076 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199215f407b7629df682'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:49:19,077 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199215f407b7629df682 2025-12-11 19:49:19,079 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199215f407b7629df682 HTTP/1.1" 502 150 2025-12-11 19:49:19,079 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199215f407b7629df682'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:49:19,515 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b140afee8383e92b742dd 2025-12-11 19:49:19,516 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b140afee8383e92b742dd HTTP/1.1" 502 150 2025-12-11 19:49:19,516 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b140afee8383e92b742dd'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:49:22,745 - lava_callback - INFO - Received callback for node 693b199315f407b7629df686 INFO: 172.21.0.1:45598 - "POST /node/693b199315f407b7629df686 HTTP/1.0" 202 Accepted 2025-12-11 19:49:22,756 - lava_callback - INFO - Starting async job submit for node 693b199315f407b7629df686 2025-12-11 19:49:22,789 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:49:22,799 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199315f407b7629df686 HTTP/1.1" 502 150 2025-12-11 19:49:22,799 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199315f407b7629df686'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:49:22,799 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199315f407b7629df686 2025-12-11 19:49:22,800 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199315f407b7629df686 HTTP/1.1" 502 150 2025-12-11 19:49:22,800 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199315f407b7629df686'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:49:23,080 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199215f407b7629df682 2025-12-11 19:49:23,081 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199215f407b7629df682 HTTP/1.1" 502 150 2025-12-11 19:49:23,081 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199215f407b7629df682'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:49:23,984 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae331fee8383e92b640d3 2025-12-11 19:49:23,985 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae331fee8383e92b640d3 HTTP/1.1" 502 150 2025-12-11 19:49:23,985 - lava_callback - ERROR - Error processing callback for node 693ae331fee8383e92b640d3: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae331fee8383e92b640d3 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae331fee8383e92b640d3 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae331fee8383e92b640d3 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:49:24,800 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199315f407b7629df686 2025-12-11 19:49:24,804 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199315f407b7629df686 HTTP/1.1" 502 150 2025-12-11 19:49:24,804 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199315f407b7629df686'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:49:25,239 - lava_callback - INFO - Received callback for node 693b11edfee8383e92b73a91 INFO: 172.21.0.1:45612 - "POST /node/693b11edfee8383e92b73a91 HTTP/1.0" 202 Accepted 2025-12-11 19:49:25,252 - lava_callback - INFO - Starting async job submit for node 693b11edfee8383e92b73a91 2025-12-11 19:49:25,316 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:49:25,330 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b11edfee8383e92b73a91 HTTP/1.1" 502 150 2025-12-11 19:49:25,330 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b11edfee8383e92b73a91'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:49:25,331 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b11edfee8383e92b73a91 2025-12-11 19:49:25,332 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b11edfee8383e92b73a91 HTTP/1.1" 502 150 2025-12-11 19:49:25,332 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b11edfee8383e92b73a91'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:45618 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45634 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:49:27,333 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b11edfee8383e92b73a91 2025-12-11 19:49:27,334 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b11edfee8383e92b73a91 HTTP/1.1" 502 150 2025-12-11 19:49:27,334 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b11edfee8383e92b73a91'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:49:28,804 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199315f407b7629df686 2025-12-11 19:49:28,805 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199315f407b7629df686 HTTP/1.1" 502 150 2025-12-11 19:49:28,805 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199315f407b7629df686'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:49:31,081 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199215f407b7629df682 2025-12-11 19:49:31,083 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199215f407b7629df682 HTTP/1.1" 502 150 2025-12-11 19:49:31,083 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199215f407b7629df682'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:49:31,334 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b11edfee8383e92b73a91 2025-12-11 19:49:31,335 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b11edfee8383e92b73a91 HTTP/1.1" 502 150 2025-12-11 19:49:31,335 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b11edfee8383e92b73a91'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:49:35,516 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b140afee8383e92b742dd 2025-12-11 19:49:35,517 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b140afee8383e92b742dd HTTP/1.1" 502 150 2025-12-11 19:49:35,517 - lava_callback - ERROR - Error processing callback for node 693b140afee8383e92b742dd: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b140afee8383e92b742dd (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b140afee8383e92b742dd (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b140afee8383e92b742dd (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:50546 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:49:36,805 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199315f407b7629df686 2025-12-11 19:49:36,807 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199315f407b7629df686 HTTP/1.1" 502 150 2025-12-11 19:49:36,807 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199315f407b7629df686'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:49:39,336 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b11edfee8383e92b73a91 2025-12-11 19:49:39,340 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b11edfee8383e92b73a91 HTTP/1.1" 502 150 2025-12-11 19:49:39,340 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b11edfee8383e92b73a91'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:49:45,387 - lava_callback - INFO - Received callback for node 693ae641fee8383e92b64c74 INFO: 172.21.0.1:38264 - "POST /node/693ae641fee8383e92b64c74 HTTP/1.0" 202 Accepted 2025-12-11 19:49:45,397 - lava_callback - INFO - Starting async job submit for node 693ae641fee8383e92b64c74 2025-12-11 19:49:45,436 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:49:45,450 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae641fee8383e92b64c74 HTTP/1.1" 502 150 2025-12-11 19:49:45,451 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae641fee8383e92b64c74'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:49:45,451 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae641fee8383e92b64c74 2025-12-11 19:49:45,452 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae641fee8383e92b64c74 HTTP/1.1" 502 150 2025-12-11 19:49:45,452 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae641fee8383e92b64c74'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:38274 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:49:47,083 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199215f407b7629df682 2025-12-11 19:49:47,084 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199215f407b7629df682 HTTP/1.1" 502 150 2025-12-11 19:49:47,084 - lava_callback - ERROR - Error processing callback for node 693b199215f407b7629df682: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b199215f407b7629df682 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b199215f407b7629df682 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b199215f407b7629df682 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:49:47,452 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae641fee8383e92b64c74 2025-12-11 19:49:47,453 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae641fee8383e92b64c74 HTTP/1.1" 502 150 2025-12-11 19:49:47,453 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae641fee8383e92b64c74'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:49:51,454 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae641fee8383e92b64c74 2025-12-11 19:49:51,455 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae641fee8383e92b64c74 HTTP/1.1" 502 150 2025-12-11 19:49:51,455 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae641fee8383e92b64c74'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:49:52,807 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199315f407b7629df686 2025-12-11 19:49:52,808 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199315f407b7629df686 HTTP/1.1" 502 150 2025-12-11 19:49:52,808 - lava_callback - ERROR - Error processing callback for node 693b199315f407b7629df686: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b199315f407b7629df686 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b199315f407b7629df686 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b199315f407b7629df686 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:49:55,340 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b11edfee8383e92b73a91 2025-12-11 19:49:55,341 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b11edfee8383e92b73a91 HTTP/1.1" 502 150 2025-12-11 19:49:55,342 - lava_callback - ERROR - Error processing callback for node 693b11edfee8383e92b73a91: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b11edfee8383e92b73a91 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b11edfee8383e92b73a91 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b11edfee8383e92b73a91 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:42208 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:49:59,455 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae641fee8383e92b64c74 2025-12-11 19:49:59,456 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae641fee8383e92b64c74 HTTP/1.1" 502 150 2025-12-11 19:49:59,456 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae641fee8383e92b64c74'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:38634 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:50:11,170 - lava_callback - INFO - Received callback for node 693afb6afee8383e92b6d480 INFO: 172.21.0.1:41668 - "POST /node/693afb6afee8383e92b6d480 HTTP/1.0" 202 Accepted 2025-12-11 19:50:11,180 - lava_callback - INFO - Starting async job submit for node 693afb6afee8383e92b6d480 2025-12-11 19:50:11,205 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:50:11,242 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afb6afee8383e92b6d480 HTTP/1.1" 502 150 2025-12-11 19:50:11,242 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afb6afee8383e92b6d480'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:11,242 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afb6afee8383e92b6d480 2025-12-11 19:50:11,244 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afb6afee8383e92b6d480 HTTP/1.1" 502 150 2025-12-11 19:50:11,244 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afb6afee8383e92b6d480'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:13,245 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afb6afee8383e92b6d480 2025-12-11 19:50:13,246 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afb6afee8383e92b6d480 HTTP/1.1" 502 150 2025-12-11 19:50:13,246 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afb6afee8383e92b6d480'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:41672 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 19:50:15,457 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae641fee8383e92b64c74 2025-12-11 19:50:15,458 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae641fee8383e92b64c74 HTTP/1.1" 502 150 2025-12-11 19:50:15,458 - lava_callback - ERROR - Error processing callback for node 693ae641fee8383e92b64c74: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae641fee8383e92b64c74 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae641fee8383e92b64c74 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae641fee8383e92b64c74 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:50:16,276 - lava_callback - INFO - Received callback for node 693b143ffee8383e92b7434a INFO: 172.21.0.1:41680 - "POST /node/693b143ffee8383e92b7434a HTTP/1.0" 202 Accepted 2025-12-11 19:50:16,287 - lava_callback - INFO - Starting async job submit for node 693b143ffee8383e92b7434a 2025-12-11 19:50:16,331 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:50:16,344 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b143ffee8383e92b7434a HTTP/1.1" 502 150 2025-12-11 19:50:16,345 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b143ffee8383e92b7434a'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:16,345 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b143ffee8383e92b7434a 2025-12-11 19:50:16,346 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b143ffee8383e92b7434a HTTP/1.1" 502 150 2025-12-11 19:50:16,346 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b143ffee8383e92b7434a'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:41692 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:50:17,246 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afb6afee8383e92b6d480 2025-12-11 19:50:17,247 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afb6afee8383e92b6d480 HTTP/1.1" 502 150 2025-12-11 19:50:17,247 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afb6afee8383e92b6d480'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:18,347 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b143ffee8383e92b7434a 2025-12-11 19:50:18,348 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b143ffee8383e92b7434a HTTP/1.1" 502 150 2025-12-11 19:50:18,348 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b143ffee8383e92b7434a'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:22,348 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b143ffee8383e92b7434a 2025-12-11 19:50:22,349 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b143ffee8383e92b7434a HTTP/1.1" 502 150 2025-12-11 19:50:22,349 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b143ffee8383e92b7434a'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:25,176 - lava_callback - INFO - Received callback for node 693af36ffee8383e92b6a951 INFO: 172.21.0.1:46142 - "POST /node/693af36ffee8383e92b6a951 HTTP/1.0" 202 Accepted 2025-12-11 19:50:25,182 - lava_callback - INFO - Starting async job submit for node 693af36ffee8383e92b6a951 2025-12-11 19:50:25,199 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:50:25,212 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af36ffee8383e92b6a951 HTTP/1.1" 502 150 2025-12-11 19:50:25,212 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af36ffee8383e92b6a951'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:25,212 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af36ffee8383e92b6a951 2025-12-11 19:50:25,212 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af36ffee8383e92b6a951 HTTP/1.1" 502 150 2025-12-11 19:50:25,213 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af36ffee8383e92b6a951'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:25,247 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afb6afee8383e92b6d480 2025-12-11 19:50:25,248 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afb6afee8383e92b6d480 HTTP/1.1" 502 150 2025-12-11 19:50:25,249 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afb6afee8383e92b6d480'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:46158 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46172 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:50:27,213 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af36ffee8383e92b6a951 2025-12-11 19:50:27,214 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af36ffee8383e92b6a951 HTTP/1.1" 502 150 2025-12-11 19:50:27,215 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af36ffee8383e92b6a951'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:30,349 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b143ffee8383e92b7434a 2025-12-11 19:50:30,351 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b143ffee8383e92b7434a HTTP/1.1" 502 150 2025-12-11 19:50:30,351 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b143ffee8383e92b7434a'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:31,215 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af36ffee8383e92b6a951 2025-12-11 19:50:31,216 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af36ffee8383e92b6a951 HTTP/1.1" 502 150 2025-12-11 19:50:31,216 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af36ffee8383e92b6a951'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:34,989 - lava_callback - INFO - Received callback for node 693b199915f407b7629df696 INFO: 172.21.0.1:46378 - "POST /node/693b199915f407b7629df696 HTTP/1.0" 202 Accepted 2025-12-11 19:50:34,999 - lava_callback - INFO - Starting async job submit for node 693b199915f407b7629df696 2025-12-11 19:50:35,038 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:50:35,051 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199915f407b7629df696 HTTP/1.1" 502 150 2025-12-11 19:50:35,052 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199915f407b7629df696'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:35,052 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199915f407b7629df696 2025-12-11 19:50:35,054 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199915f407b7629df696 HTTP/1.1" 502 150 2025-12-11 19:50:35,054 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199915f407b7629df696'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:46394 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:50:37,054 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199915f407b7629df696 2025-12-11 19:50:37,056 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199915f407b7629df696 HTTP/1.1" 502 150 2025-12-11 19:50:37,056 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199915f407b7629df696'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:39,216 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af36ffee8383e92b6a951 2025-12-11 19:50:39,217 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af36ffee8383e92b6a951 HTTP/1.1" 502 150 2025-12-11 19:50:39,217 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af36ffee8383e92b6a951'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:41,056 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199915f407b7629df696 2025-12-11 19:50:41,057 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199915f407b7629df696 HTTP/1.1" 502 150 2025-12-11 19:50:41,057 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199915f407b7629df696'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:41,249 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afb6afee8383e92b6d480 2025-12-11 19:50:41,250 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afb6afee8383e92b6d480 HTTP/1.1" 502 150 2025-12-11 19:50:41,250 - lava_callback - ERROR - Error processing callback for node 693afb6afee8383e92b6d480: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afb6afee8383e92b6d480 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afb6afee8383e92b6d480 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afb6afee8383e92b6d480 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:50:45,748 - lava_callback - INFO - Received callback for node 693b0760fee8383e92b70dcb INFO: 172.21.0.1:53680 - "POST /node/693b0760fee8383e92b70dcb HTTP/1.0" 202 Accepted 2025-12-11 19:50:45,757 - lava_callback - INFO - Starting async job submit for node 693b0760fee8383e92b70dcb 2025-12-11 19:50:45,807 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:50:45,821 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0760fee8383e92b70dcb HTTP/1.1" 502 150 2025-12-11 19:50:45,821 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0760fee8383e92b70dcb'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:45,821 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0760fee8383e92b70dcb 2025-12-11 19:50:45,821 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0760fee8383e92b70dcb HTTP/1.1" 502 150 2025-12-11 19:50:45,821 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0760fee8383e92b70dcb'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:46,046 - lava_callback - INFO - Received callback for node 693ae850fee8383e92b6589c INFO: 172.21.0.1:53684 - "POST /node/693ae850fee8383e92b6589c HTTP/1.0" 202 Accepted 2025-12-11 19:50:46,053 - lava_callback - INFO - Starting async job submit for node 693ae850fee8383e92b6589c 2025-12-11 19:50:46,084 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:50:46,095 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae850fee8383e92b6589c HTTP/1.1" 502 150 2025-12-11 19:50:46,095 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae850fee8383e92b6589c'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:46,096 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae850fee8383e92b6589c 2025-12-11 19:50:46,096 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae850fee8383e92b6589c HTTP/1.1" 502 150 2025-12-11 19:50:46,096 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae850fee8383e92b6589c'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:46,351 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b143ffee8383e92b7434a 2025-12-11 19:50:46,352 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b143ffee8383e92b7434a HTTP/1.1" 502 150 2025-12-11 19:50:46,352 - lava_callback - ERROR - Error processing callback for node 693b143ffee8383e92b7434a: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b143ffee8383e92b7434a (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b143ffee8383e92b7434a (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b143ffee8383e92b7434a (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:53698 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:50:47,822 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0760fee8383e92b70dcb 2025-12-11 19:50:47,825 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0760fee8383e92b70dcb HTTP/1.1" 502 150 2025-12-11 19:50:47,825 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0760fee8383e92b70dcb'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:48,097 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae850fee8383e92b6589c 2025-12-11 19:50:48,098 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae850fee8383e92b6589c HTTP/1.1" 502 150 2025-12-11 19:50:48,098 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae850fee8383e92b6589c'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:49,058 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199915f407b7629df696 2025-12-11 19:50:49,060 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199915f407b7629df696 HTTP/1.1" 502 150 2025-12-11 19:50:49,060 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199915f407b7629df696'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:49,226 - lava_callback - INFO - Received callback for node 693ae594fee8383e92b64931 INFO: 172.21.0.1:53700 - "POST /node/693ae594fee8383e92b64931 HTTP/1.0" 202 Accepted 2025-12-11 19:50:49,232 - lava_callback - INFO - Starting async job submit for node 693ae594fee8383e92b64931 2025-12-11 19:50:49,275 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:50:49,295 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae594fee8383e92b64931 HTTP/1.1" 502 150 2025-12-11 19:50:49,295 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae594fee8383e92b64931'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:49,295 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae594fee8383e92b64931 2025-12-11 19:50:49,296 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae594fee8383e92b64931 HTTP/1.1" 502 150 2025-12-11 19:50:49,296 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae594fee8383e92b64931'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:51,296 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae594fee8383e92b64931 2025-12-11 19:50:51,297 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae594fee8383e92b64931 HTTP/1.1" 502 150 2025-12-11 19:50:51,297 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae594fee8383e92b64931'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:51,825 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0760fee8383e92b70dcb 2025-12-11 19:50:51,827 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0760fee8383e92b70dcb HTTP/1.1" 502 150 2025-12-11 19:50:51,827 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0760fee8383e92b70dcb'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:52,098 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae850fee8383e92b6589c 2025-12-11 19:50:52,099 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae850fee8383e92b6589c HTTP/1.1" 502 150 2025-12-11 19:50:52,099 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae850fee8383e92b6589c'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:50:55,218 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af36ffee8383e92b6a951 2025-12-11 19:50:55,224 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af36ffee8383e92b6a951 HTTP/1.1" 502 150 2025-12-11 19:50:55,224 - lava_callback - ERROR - Error processing callback for node 693af36ffee8383e92b6a951: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af36ffee8383e92b6a951 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af36ffee8383e92b6a951 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af36ffee8383e92b6a951 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:50:55,297 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae594fee8383e92b64931 2025-12-11 19:50:55,298 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae594fee8383e92b64931 HTTP/1.1" 502 150 2025-12-11 19:50:55,298 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae594fee8383e92b64931'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:45100 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:50:59,827 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0760fee8383e92b70dcb 2025-12-11 19:50:59,828 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0760fee8383e92b70dcb HTTP/1.1" 502 150 2025-12-11 19:50:59,828 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0760fee8383e92b70dcb'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:00,100 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae850fee8383e92b6589c 2025-12-11 19:51:00,101 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae850fee8383e92b6589c HTTP/1.1" 502 150 2025-12-11 19:51:00,101 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae850fee8383e92b6589c'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:03,299 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae594fee8383e92b64931 2025-12-11 19:51:03,300 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae594fee8383e92b64931 HTTP/1.1" 502 150 2025-12-11 19:51:03,300 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae594fee8383e92b64931'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:05,060 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199915f407b7629df696 2025-12-11 19:51:05,062 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199915f407b7629df696 HTTP/1.1" 502 150 2025-12-11 19:51:05,062 - lava_callback - ERROR - Error processing callback for node 693b199915f407b7629df696: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b199915f407b7629df696 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b199915f407b7629df696 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b199915f407b7629df696 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:46514 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54788 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 19:51:15,829 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0760fee8383e92b70dcb 2025-12-11 19:51:15,830 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0760fee8383e92b70dcb HTTP/1.1" 502 150 2025-12-11 19:51:15,830 - lava_callback - ERROR - Error processing callback for node 693b0760fee8383e92b70dcb: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0760fee8383e92b70dcb (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0760fee8383e92b70dcb (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0760fee8383e92b70dcb (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:51:16,102 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae850fee8383e92b6589c 2025-12-11 19:51:16,103 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae850fee8383e92b6589c HTTP/1.1" 502 150 2025-12-11 19:51:16,103 - lava_callback - ERROR - Error processing callback for node 693ae850fee8383e92b6589c: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae850fee8383e92b6589c (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae850fee8383e92b6589c (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae850fee8383e92b6589c (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:54798 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:51:18,717 - lava_callback - INFO - Received callback for node 693b199915f407b7629df698 INFO: 172.21.0.1:54812 - "POST /node/693b199915f407b7629df698 HTTP/1.0" 202 Accepted 2025-12-11 19:51:18,723 - lava_callback - INFO - Starting async job submit for node 693b199915f407b7629df698 2025-12-11 19:51:18,750 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:51:18,762 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199915f407b7629df698 HTTP/1.1" 502 150 2025-12-11 19:51:18,762 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199915f407b7629df698'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:18,763 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199915f407b7629df698 2025-12-11 19:51:18,763 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199915f407b7629df698 HTTP/1.1" 502 150 2025-12-11 19:51:18,764 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199915f407b7629df698'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:19,300 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae594fee8383e92b64931 2025-12-11 19:51:19,302 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae594fee8383e92b64931 HTTP/1.1" 502 150 2025-12-11 19:51:19,302 - lava_callback - ERROR - Error processing callback for node 693ae594fee8383e92b64931: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae594fee8383e92b64931 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae594fee8383e92b64931 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae594fee8383e92b64931 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:51:20,764 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199915f407b7629df698 2025-12-11 19:51:20,765 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199915f407b7629df698 HTTP/1.1" 502 150 2025-12-11 19:51:20,765 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199915f407b7629df698'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:24,765 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199915f407b7629df698 2025-12-11 19:51:24,767 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199915f407b7629df698 HTTP/1.1" 502 150 2025-12-11 19:51:24,767 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199915f407b7629df698'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:25,305 - lava_callback - INFO - Received callback for node 693b09d3fee8383e92b718d8 INFO: 172.21.0.1:51264 - "POST /node/693b09d3fee8383e92b718d8 HTTP/1.0" 202 Accepted 2025-12-11 19:51:25,311 - lava_callback - INFO - Starting async job submit for node 693b09d3fee8383e92b718d8 2025-12-11 19:51:25,359 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:51:25,370 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b09d3fee8383e92b718d8 HTTP/1.1" 502 150 2025-12-11 19:51:25,370 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b09d3fee8383e92b718d8'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:25,370 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b09d3fee8383e92b718d8 2025-12-11 19:51:25,371 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b09d3fee8383e92b718d8 HTTP/1.1" 502 150 2025-12-11 19:51:25,371 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b09d3fee8383e92b718d8'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:26,475 - lava_callback - INFO - Received callback for node 693ae845fee8383e92b65867 INFO: 172.21.0.1:51270 - "POST /node/693ae845fee8383e92b65867 HTTP/1.0" 202 Accepted 2025-12-11 19:51:26,483 - lava_callback - INFO - Starting async job submit for node 693ae845fee8383e92b65867 INFO: 172.21.0.1:51286 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:51:26,656 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:51:26,666 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae845fee8383e92b65867 HTTP/1.1" 502 150 2025-12-11 19:51:26,666 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae845fee8383e92b65867'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:26,667 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae845fee8383e92b65867 2025-12-11 19:51:26,667 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae845fee8383e92b65867 HTTP/1.1" 502 150 2025-12-11 19:51:26,668 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae845fee8383e92b65867'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:51296 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:51:27,371 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b09d3fee8383e92b718d8 2025-12-11 19:51:27,373 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b09d3fee8383e92b718d8 HTTP/1.1" 502 150 2025-12-11 19:51:27,373 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b09d3fee8383e92b718d8'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:28,060 - lava_callback - INFO - Received callback for node 693ae851fee8383e92b658a1 INFO: 172.21.0.1:51312 - "POST /node/693ae851fee8383e92b658a1 HTTP/1.0" 202 Accepted 2025-12-11 19:51:28,067 - lava_callback - INFO - Starting async job submit for node 693ae851fee8383e92b658a1 2025-12-11 19:51:28,118 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:51:28,134 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae851fee8383e92b658a1 HTTP/1.1" 502 150 2025-12-11 19:51:28,134 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae851fee8383e92b658a1'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:28,134 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae851fee8383e92b658a1 2025-12-11 19:51:28,135 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae851fee8383e92b658a1 HTTP/1.1" 502 150 2025-12-11 19:51:28,135 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae851fee8383e92b658a1'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:28,668 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae845fee8383e92b65867 2025-12-11 19:51:28,675 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae845fee8383e92b65867 HTTP/1.1" 502 150 2025-12-11 19:51:28,676 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae845fee8383e92b65867'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:30,135 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae851fee8383e92b658a1 2025-12-11 19:51:30,137 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae851fee8383e92b658a1 HTTP/1.1" 502 150 2025-12-11 19:51:30,137 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae851fee8383e92b658a1'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:31,373 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b09d3fee8383e92b718d8 2025-12-11 19:51:31,374 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b09d3fee8383e92b718d8 HTTP/1.1" 502 150 2025-12-11 19:51:31,374 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b09d3fee8383e92b718d8'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:32,676 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae845fee8383e92b65867 2025-12-11 19:51:32,678 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae845fee8383e92b65867 HTTP/1.1" 502 150 2025-12-11 19:51:32,678 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae845fee8383e92b65867'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:32,767 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199915f407b7629df698 2025-12-11 19:51:32,768 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199915f407b7629df698 HTTP/1.1" 502 150 2025-12-11 19:51:32,768 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199915f407b7629df698'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:34,137 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae851fee8383e92b658a1 2025-12-11 19:51:34,138 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae851fee8383e92b658a1 HTTP/1.1" 502 150 2025-12-11 19:51:34,138 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae851fee8383e92b658a1'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:40674 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:51:39,374 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b09d3fee8383e92b718d8 2025-12-11 19:51:39,376 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b09d3fee8383e92b718d8 HTTP/1.1" 502 150 2025-12-11 19:51:39,376 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b09d3fee8383e92b718d8'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:40,678 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae845fee8383e92b65867 2025-12-11 19:51:40,679 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae845fee8383e92b65867 HTTP/1.1" 502 150 2025-12-11 19:51:40,679 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae845fee8383e92b65867'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:42,138 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae851fee8383e92b658a1 2025-12-11 19:51:42,140 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae851fee8383e92b658a1 HTTP/1.1" 502 150 2025-12-11 19:51:42,140 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae851fee8383e92b658a1'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:43,045 - lava_callback - INFO - Received callback for node 693b199e15f407b7629df6a4 INFO: 172.21.0.1:45506 - "POST /node/693b199e15f407b7629df6a4 HTTP/1.0" 202 Accepted 2025-12-11 19:51:43,052 - lava_callback - INFO - Starting async job submit for node 693b199e15f407b7629df6a4 2025-12-11 19:51:43,076 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:51:43,111 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199e15f407b7629df6a4 HTTP/1.1" 502 150 2025-12-11 19:51:43,111 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199e15f407b7629df6a4'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:43,111 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199e15f407b7629df6a4 2025-12-11 19:51:43,112 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199e15f407b7629df6a4 HTTP/1.1" 502 150 2025-12-11 19:51:43,112 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199e15f407b7629df6a4'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:45,112 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199e15f407b7629df6a4 2025-12-11 19:51:45,113 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199e15f407b7629df6a4 HTTP/1.1" 502 150 2025-12-11 19:51:45,113 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199e15f407b7629df6a4'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:45,431 - lava_callback - INFO - Received callback for node 693ae642fee8383e92b64c76 INFO: 172.21.0.1:45516 - "POST /node/693ae642fee8383e92b64c76 HTTP/1.0" 202 Accepted 2025-12-11 19:51:45,440 - lava_callback - INFO - Starting async job submit for node 693ae642fee8383e92b64c76 2025-12-11 19:51:45,468 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:51:45,478 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae642fee8383e92b64c76 HTTP/1.1" 502 150 2025-12-11 19:51:45,479 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae642fee8383e92b64c76'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:45,479 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae642fee8383e92b64c76 2025-12-11 19:51:45,480 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae642fee8383e92b64c76 HTTP/1.1" 502 150 2025-12-11 19:51:45,480 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae642fee8383e92b64c76'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:45524 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:51:47,068 - lava_callback - INFO - Received callback for node 693ae852fee8383e92b658a6 INFO: 172.21.0.1:45538 - "POST /node/693ae852fee8383e92b658a6 HTTP/1.0" 202 Accepted 2025-12-11 19:51:47,075 - lava_callback - INFO - Starting async job submit for node 693ae852fee8383e92b658a6 2025-12-11 19:51:47,101 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:51:47,128 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae852fee8383e92b658a6 HTTP/1.1" 502 150 2025-12-11 19:51:47,128 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae852fee8383e92b658a6'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:47,128 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae852fee8383e92b658a6 2025-12-11 19:51:47,130 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae852fee8383e92b658a6 HTTP/1.1" 502 150 2025-12-11 19:51:47,130 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae852fee8383e92b658a6'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:47,480 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae642fee8383e92b64c76 2025-12-11 19:51:47,481 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae642fee8383e92b64c76 HTTP/1.1" 502 150 2025-12-11 19:51:47,481 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae642fee8383e92b64c76'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:48,769 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199915f407b7629df698 2025-12-11 19:51:48,770 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199915f407b7629df698 HTTP/1.1" 502 150 2025-12-11 19:51:48,770 - lava_callback - ERROR - Error processing callback for node 693b199915f407b7629df698: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b199915f407b7629df698 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b199915f407b7629df698 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b199915f407b7629df698 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:51:49,113 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199e15f407b7629df6a4 2025-12-11 19:51:49,115 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199e15f407b7629df6a4 HTTP/1.1" 502 150 2025-12-11 19:51:49,115 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199e15f407b7629df6a4'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:49,130 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae852fee8383e92b658a6 2025-12-11 19:51:49,131 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae852fee8383e92b658a6 HTTP/1.1" 502 150 2025-12-11 19:51:49,131 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae852fee8383e92b658a6'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:51,481 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae642fee8383e92b64c76 2025-12-11 19:51:51,482 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae642fee8383e92b64c76 HTTP/1.1" 502 150 2025-12-11 19:51:51,482 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae642fee8383e92b64c76'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:53,131 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae852fee8383e92b658a6 2025-12-11 19:51:53,133 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae852fee8383e92b658a6 HTTP/1.1" 502 150 2025-12-11 19:51:53,133 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae852fee8383e92b658a6'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:55,376 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b09d3fee8383e92b718d8 2025-12-11 19:51:55,378 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b09d3fee8383e92b718d8 HTTP/1.1" 502 150 2025-12-11 19:51:55,378 - lava_callback - ERROR - Error processing callback for node 693b09d3fee8383e92b718d8: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b09d3fee8383e92b718d8 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b09d3fee8383e92b718d8 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b09d3fee8383e92b718d8 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:50446 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:51:56,679 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae845fee8383e92b65867 2025-12-11 19:51:56,680 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae845fee8383e92b65867 HTTP/1.1" 502 150 2025-12-11 19:51:56,681 - lava_callback - ERROR - Error processing callback for node 693ae845fee8383e92b65867: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae845fee8383e92b65867 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae845fee8383e92b65867 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae845fee8383e92b65867 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:51:57,115 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199e15f407b7629df6a4 2025-12-11 19:51:57,116 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199e15f407b7629df6a4 HTTP/1.1" 502 150 2025-12-11 19:51:57,116 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b199e15f407b7629df6a4'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:51:58,140 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae851fee8383e92b658a1 2025-12-11 19:51:58,141 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae851fee8383e92b658a1 HTTP/1.1" 502 150 2025-12-11 19:51:58,142 - lava_callback - ERROR - Error processing callback for node 693ae851fee8383e92b658a1: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae851fee8383e92b658a1 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae851fee8383e92b658a1 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae851fee8383e92b658a1 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:51:59,483 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae642fee8383e92b64c76 2025-12-11 19:51:59,484 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae642fee8383e92b64c76 HTTP/1.1" 502 150 2025-12-11 19:51:59,484 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae642fee8383e92b64c76'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:01,059 - lava_callback - INFO - Received callback for node 693b078dfee8383e92b70ecc INFO: 172.21.0.1:36760 - "POST /node/693b078dfee8383e92b70ecc HTTP/1.0" 202 Accepted 2025-12-11 19:52:01,071 - lava_callback - INFO - Starting async job submit for node 693b078dfee8383e92b70ecc 2025-12-11 19:52:01,099 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:52:01,114 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b078dfee8383e92b70ecc HTTP/1.1" 502 150 2025-12-11 19:52:01,114 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b078dfee8383e92b70ecc'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:01,114 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b078dfee8383e92b70ecc 2025-12-11 19:52:01,115 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b078dfee8383e92b70ecc HTTP/1.1" 502 150 2025-12-11 19:52:01,115 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b078dfee8383e92b70ecc'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:01,133 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae852fee8383e92b658a6 2025-12-11 19:52:01,134 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae852fee8383e92b658a6 HTTP/1.1" 502 150 2025-12-11 19:52:01,134 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae852fee8383e92b658a6'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:03,115 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b078dfee8383e92b70ecc 2025-12-11 19:52:03,116 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b078dfee8383e92b70ecc HTTP/1.1" 502 150 2025-12-11 19:52:03,116 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b078dfee8383e92b70ecc'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:04,109 - lava_callback - INFO - Received callback for node 693b17e915f407b7629df38a INFO: 172.21.0.1:36776 - "POST /node/693b17e915f407b7629df38a HTTP/1.0" 202 Accepted 2025-12-11 19:52:04,117 - lava_callback - INFO - Starting async job submit for node 693b17e915f407b7629df38a 2025-12-11 19:52:04,153 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:52:04,163 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b17e915f407b7629df38a HTTP/1.1" 502 150 2025-12-11 19:52:04,163 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b17e915f407b7629df38a'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:04,164 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b17e915f407b7629df38a 2025-12-11 19:52:04,164 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b17e915f407b7629df38a HTTP/1.1" 502 150 2025-12-11 19:52:04,164 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b17e915f407b7629df38a'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:06,165 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b17e915f407b7629df38a 2025-12-11 19:52:06,166 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b17e915f407b7629df38a HTTP/1.1" 502 150 2025-12-11 19:52:06,166 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b17e915f407b7629df38a'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:36784 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:52:07,116 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b078dfee8383e92b70ecc 2025-12-11 19:52:07,117 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b078dfee8383e92b70ecc HTTP/1.1" 502 150 2025-12-11 19:52:07,118 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b078dfee8383e92b70ecc'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:10,167 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b17e915f407b7629df38a 2025-12-11 19:52:10,168 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b17e915f407b7629df38a HTTP/1.1" 502 150 2025-12-11 19:52:10,168 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b17e915f407b7629df38a'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:13,116 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b199e15f407b7629df6a4 2025-12-11 19:52:13,118 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b199e15f407b7629df6a4 HTTP/1.1" 502 150 2025-12-11 19:52:13,118 - lava_callback - ERROR - Error processing callback for node 693b199e15f407b7629df6a4: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b199e15f407b7629df6a4 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b199e15f407b7629df6a4 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b199e15f407b7629df6a4 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:52:13,676 - lava_callback - INFO - Received callback for node 693b1a6315f407b7629dfbfe INFO: 172.21.0.1:39824 - "POST /node/693b1a6315f407b7629dfbfe HTTP/1.0" 202 Accepted 2025-12-11 19:52:13,683 - lava_callback - INFO - Starting async job submit for node 693b1a6315f407b7629dfbfe 2025-12-11 19:52:13,705 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:52:13,715 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a6315f407b7629dfbfe HTTP/1.1" 502 150 2025-12-11 19:52:13,715 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a6315f407b7629dfbfe'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:13,715 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a6315f407b7629dfbfe 2025-12-11 19:52:13,715 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a6315f407b7629dfbfe HTTP/1.1" 502 150 2025-12-11 19:52:13,716 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a6315f407b7629dfbfe'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:15,118 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b078dfee8383e92b70ecc 2025-12-11 19:52:15,119 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b078dfee8383e92b70ecc HTTP/1.1" 502 150 2025-12-11 19:52:15,119 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b078dfee8383e92b70ecc'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:39832 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 19:52:15,484 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae642fee8383e92b64c76 2025-12-11 19:52:15,485 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae642fee8383e92b64c76 HTTP/1.1" 502 150 2025-12-11 19:52:15,486 - lava_callback - ERROR - Error processing callback for node 693ae642fee8383e92b64c76: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae642fee8383e92b64c76 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae642fee8383e92b64c76 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae642fee8383e92b64c76 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:52:15,716 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a6315f407b7629dfbfe 2025-12-11 19:52:15,717 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a6315f407b7629dfbfe HTTP/1.1" 502 150 2025-12-11 19:52:15,717 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a6315f407b7629dfbfe'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:39836 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:52:17,134 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae852fee8383e92b658a6 2025-12-11 19:52:17,136 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae852fee8383e92b658a6 HTTP/1.1" 502 150 2025-12-11 19:52:17,136 - lava_callback - ERROR - Error processing callback for node 693ae852fee8383e92b658a6: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae852fee8383e92b658a6 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae852fee8383e92b658a6 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae852fee8383e92b658a6 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:52:18,168 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b17e915f407b7629df38a 2025-12-11 19:52:18,169 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b17e915f407b7629df38a HTTP/1.1" 502 150 2025-12-11 19:52:18,169 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b17e915f407b7629df38a'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:19,442 - lava_callback - INFO - Received callback for node 693b1445fee8383e92b7435c INFO: 172.21.0.1:39844 - "POST /node/693b1445fee8383e92b7435c HTTP/1.0" 202 Accepted 2025-12-11 19:52:19,448 - lava_callback - INFO - Starting async job submit for node 693b1445fee8383e92b7435c 2025-12-11 19:52:19,467 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:52:19,482 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1445fee8383e92b7435c HTTP/1.1" 502 150 2025-12-11 19:52:19,482 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1445fee8383e92b7435c'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:19,482 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1445fee8383e92b7435c 2025-12-11 19:52:19,483 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1445fee8383e92b7435c HTTP/1.1" 502 150 2025-12-11 19:52:19,483 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1445fee8383e92b7435c'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:19,717 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a6315f407b7629dfbfe 2025-12-11 19:52:19,718 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a6315f407b7629dfbfe HTTP/1.1" 502 150 2025-12-11 19:52:19,719 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a6315f407b7629dfbfe'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:21,483 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1445fee8383e92b7435c 2025-12-11 19:52:21,484 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1445fee8383e92b7435c HTTP/1.1" 502 150 2025-12-11 19:52:21,484 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1445fee8383e92b7435c'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:25,485 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1445fee8383e92b7435c 2025-12-11 19:52:25,486 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1445fee8383e92b7435c HTTP/1.1" 502 150 2025-12-11 19:52:25,486 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1445fee8383e92b7435c'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:42126 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42134 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:52:27,719 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a6315f407b7629dfbfe 2025-12-11 19:52:27,720 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a6315f407b7629dfbfe HTTP/1.1" 502 150 2025-12-11 19:52:27,720 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a6315f407b7629dfbfe'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:28,184 - lava_callback - INFO - Received callback for node 693ae595fee8383e92b64934 INFO: 172.21.0.1:42144 - "POST /node/693ae595fee8383e92b64934 HTTP/1.0" 202 Accepted 2025-12-11 19:52:28,194 - lava_callback - INFO - Starting async job submit for node 693ae595fee8383e92b64934 2025-12-11 19:52:28,219 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:52:28,239 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae595fee8383e92b64934 HTTP/1.1" 502 150 2025-12-11 19:52:28,239 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae595fee8383e92b64934'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:28,239 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae595fee8383e92b64934 2025-12-11 19:52:28,240 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae595fee8383e92b64934 HTTP/1.1" 502 150 2025-12-11 19:52:28,240 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae595fee8383e92b64934'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:30,240 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae595fee8383e92b64934 2025-12-11 19:52:30,241 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae595fee8383e92b64934 HTTP/1.1" 502 150 2025-12-11 19:52:30,241 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae595fee8383e92b64934'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:31,119 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b078dfee8383e92b70ecc 2025-12-11 19:52:31,120 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b078dfee8383e92b70ecc HTTP/1.1" 502 150 2025-12-11 19:52:31,121 - lava_callback - ERROR - Error processing callback for node 693b078dfee8383e92b70ecc: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b078dfee8383e92b70ecc (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b078dfee8383e92b70ecc (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b078dfee8383e92b70ecc (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:52:33,486 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1445fee8383e92b7435c 2025-12-11 19:52:33,488 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1445fee8383e92b7435c HTTP/1.1" 502 150 2025-12-11 19:52:33,488 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1445fee8383e92b7435c'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:34,170 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b17e915f407b7629df38a 2025-12-11 19:52:34,171 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b17e915f407b7629df38a HTTP/1.1" 502 150 2025-12-11 19:52:34,171 - lava_callback - ERROR - Error processing callback for node 693b17e915f407b7629df38a: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b17e915f407b7629df38a (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b17e915f407b7629df38a (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b17e915f407b7629df38a (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:52:34,242 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae595fee8383e92b64934 2025-12-11 19:52:34,242 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae595fee8383e92b64934 HTTP/1.1" 502 150 2025-12-11 19:52:34,243 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae595fee8383e92b64934'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:34,932 - lava_callback - INFO - Received callback for node 693b1a6315f407b7629dfbfc INFO: 172.21.0.1:39958 - "POST /node/693b1a6315f407b7629dfbfc HTTP/1.0" 202 Accepted 2025-12-11 19:52:34,939 - lava_callback - INFO - Starting async job submit for node 693b1a6315f407b7629dfbfc 2025-12-11 19:52:34,961 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:52:34,973 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a6315f407b7629dfbfc HTTP/1.1" 502 150 2025-12-11 19:52:34,974 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a6315f407b7629dfbfc'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:34,974 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a6315f407b7629dfbfc 2025-12-11 19:52:34,974 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a6315f407b7629dfbfc HTTP/1.1" 502 150 2025-12-11 19:52:34,975 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a6315f407b7629dfbfc'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:39966 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:52:36,975 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a6315f407b7629dfbfc 2025-12-11 19:52:36,976 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a6315f407b7629dfbfc HTTP/1.1" 502 150 2025-12-11 19:52:36,976 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a6315f407b7629dfbfc'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:40,976 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a6315f407b7629dfbfc 2025-12-11 19:52:40,978 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a6315f407b7629dfbfc HTTP/1.1" 502 150 2025-12-11 19:52:40,978 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a6315f407b7629dfbfc'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:42,243 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae595fee8383e92b64934 2025-12-11 19:52:42,244 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae595fee8383e92b64934 HTTP/1.1" 502 150 2025-12-11 19:52:42,244 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae595fee8383e92b64934'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:43,720 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a6315f407b7629dfbfe 2025-12-11 19:52:43,721 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a6315f407b7629dfbfe HTTP/1.1" 502 150 2025-12-11 19:52:43,722 - lava_callback - ERROR - Error processing callback for node 693b1a6315f407b7629dfbfe: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1a6315f407b7629dfbfe (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1a6315f407b7629dfbfe (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1a6315f407b7629dfbfe (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:44238 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:52:48,978 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a6315f407b7629dfbfc 2025-12-11 19:52:48,980 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a6315f407b7629dfbfc HTTP/1.1" 502 150 2025-12-11 19:52:48,980 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a6315f407b7629dfbfc'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:49,488 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1445fee8383e92b7435c 2025-12-11 19:52:49,489 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1445fee8383e92b7435c HTTP/1.1" 502 150 2025-12-11 19:52:49,490 - lava_callback - ERROR - Error processing callback for node 693b1445fee8383e92b7435c: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1445fee8383e92b7435c (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1445fee8383e92b7435c (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1445fee8383e92b7435c (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:35052 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:52:57,590 - lava_callback - INFO - Received callback for node 693afccbfee8383e92b6daa8 INFO: 172.21.0.1:35068 - "POST /node/693afccbfee8383e92b6daa8 HTTP/1.0" 202 Accepted 2025-12-11 19:52:57,601 - lava_callback - INFO - Starting async job submit for node 693afccbfee8383e92b6daa8 2025-12-11 19:52:57,660 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:52:57,671 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afccbfee8383e92b6daa8 HTTP/1.1" 502 150 2025-12-11 19:52:57,671 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afccbfee8383e92b6daa8'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:57,671 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afccbfee8383e92b6daa8 2025-12-11 19:52:57,672 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afccbfee8383e92b6daa8 HTTP/1.1" 502 150 2025-12-11 19:52:57,672 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afccbfee8383e92b6daa8'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:58,244 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae595fee8383e92b64934 2025-12-11 19:52:58,246 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae595fee8383e92b64934 HTTP/1.1" 502 150 2025-12-11 19:52:58,246 - lava_callback - ERROR - Error processing callback for node 693ae595fee8383e92b64934: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae595fee8383e92b64934 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae595fee8383e92b64934 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae595fee8383e92b64934 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:52:58,733 - lava_callback - INFO - Received callback for node 693b1a6815f407b7629dfc12 INFO: 172.21.0.1:35076 - "POST /node/693b1a6815f407b7629dfc12 HTTP/1.0" 202 Accepted 2025-12-11 19:52:58,740 - lava_callback - INFO - Starting async job submit for node 693b1a6815f407b7629dfc12 2025-12-11 19:52:58,762 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:52:58,772 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a6815f407b7629dfc12 HTTP/1.1" 502 150 2025-12-11 19:52:58,772 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a6815f407b7629dfc12'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:58,772 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a6815f407b7629dfc12 2025-12-11 19:52:58,773 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a6815f407b7629dfc12 HTTP/1.1" 502 150 2025-12-11 19:52:58,773 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a6815f407b7629dfc12'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:52:59,672 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afccbfee8383e92b6daa8 2025-12-11 19:52:59,673 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afccbfee8383e92b6daa8 HTTP/1.1" 502 150 2025-12-11 19:52:59,673 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afccbfee8383e92b6daa8'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:53:00,773 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a6815f407b7629dfc12 2025-12-11 19:53:00,774 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a6815f407b7629dfc12 HTTP/1.1" 502 150 2025-12-11 19:53:00,774 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a6815f407b7629dfc12'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:53:03,674 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afccbfee8383e92b6daa8 2025-12-11 19:53:03,675 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afccbfee8383e92b6daa8 HTTP/1.1" 502 150 2025-12-11 19:53:03,675 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afccbfee8383e92b6daa8'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:53:04,775 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a6815f407b7629dfc12 2025-12-11 19:53:04,776 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a6815f407b7629dfc12 HTTP/1.1" 502 150 2025-12-11 19:53:04,776 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a6815f407b7629dfc12'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:53:04,980 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a6315f407b7629dfbfc 2025-12-11 19:53:04,981 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a6315f407b7629dfbfc HTTP/1.1" 502 150 2025-12-11 19:53:04,981 - lava_callback - ERROR - Error processing callback for node 693b1a6315f407b7629dfbfc: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1a6315f407b7629dfbfc (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1a6315f407b7629dfbfc (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1a6315f407b7629dfbfc (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:56366 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:53:11,675 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afccbfee8383e92b6daa8 2025-12-11 19:53:11,676 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afccbfee8383e92b6daa8 HTTP/1.1" 502 150 2025-12-11 19:53:11,676 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afccbfee8383e92b6daa8'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:53:12,776 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a6815f407b7629dfc12 2025-12-11 19:53:12,777 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a6815f407b7629dfc12 HTTP/1.1" 502 150 2025-12-11 19:53:12,777 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a6815f407b7629dfc12'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:37208 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 19:53:16,130 - lava_callback - INFO - Received callback for node 693b05ebfee8383e92b704ac INFO: 172.21.0.1:37224 - "POST /node/693b05ebfee8383e92b704ac HTTP/1.0" 202 Accepted 2025-12-11 19:53:16,145 - lava_callback - INFO - Starting async job submit for node 693b05ebfee8383e92b704ac 2025-12-11 19:53:16,179 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:53:16,192 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b05ebfee8383e92b704ac HTTP/1.1" 502 150 2025-12-11 19:53:16,192 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b05ebfee8383e92b704ac'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:53:16,192 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b05ebfee8383e92b704ac 2025-12-11 19:53:16,193 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b05ebfee8383e92b704ac HTTP/1.1" 502 150 2025-12-11 19:53:16,193 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b05ebfee8383e92b704ac'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:37238 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:53:18,194 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b05ebfee8383e92b704ac 2025-12-11 19:53:18,195 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b05ebfee8383e92b704ac HTTP/1.1" 502 150 2025-12-11 19:53:18,195 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b05ebfee8383e92b704ac'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:53:22,195 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b05ebfee8383e92b704ac 2025-12-11 19:53:22,196 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b05ebfee8383e92b704ac HTTP/1.1" 502 150 2025-12-11 19:53:22,196 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b05ebfee8383e92b704ac'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:53:25,573 - lava_callback - INFO - Received callback for node 693b04d2fee8383e92b6f99c INFO: 172.21.0.1:57368 - "POST /node/693b04d2fee8383e92b6f99c HTTP/1.0" 202 Accepted 2025-12-11 19:53:25,583 - lava_callback - INFO - Starting async job submit for node 693b04d2fee8383e92b6f99c 2025-12-11 19:53:25,646 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:53:25,656 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b04d2fee8383e92b6f99c HTTP/1.1" 502 150 2025-12-11 19:53:25,656 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b04d2fee8383e92b6f99c'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:53:25,656 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b04d2fee8383e92b6f99c 2025-12-11 19:53:25,657 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b04d2fee8383e92b6f99c HTTP/1.1" 502 150 2025-12-11 19:53:25,657 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b04d2fee8383e92b6f99c'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:57374 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57376 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:53:27,657 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b04d2fee8383e92b6f99c 2025-12-11 19:53:27,658 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b04d2fee8383e92b6f99c HTTP/1.1" 502 150 2025-12-11 19:53:27,658 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b04d2fee8383e92b6f99c'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:53:27,677 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afccbfee8383e92b6daa8 2025-12-11 19:53:27,679 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afccbfee8383e92b6daa8 HTTP/1.1" 502 150 2025-12-11 19:53:27,679 - lava_callback - ERROR - Error processing callback for node 693afccbfee8383e92b6daa8: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afccbfee8383e92b6daa8 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afccbfee8383e92b6daa8 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afccbfee8383e92b6daa8 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:53:28,778 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a6815f407b7629dfc12 2025-12-11 19:53:28,779 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a6815f407b7629dfc12 HTTP/1.1" 502 150 2025-12-11 19:53:28,779 - lava_callback - ERROR - Error processing callback for node 693b1a6815f407b7629dfc12: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1a6815f407b7629dfc12 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1a6815f407b7629dfc12 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1a6815f407b7629dfc12 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:53:30,196 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b05ebfee8383e92b704ac 2025-12-11 19:53:30,197 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b05ebfee8383e92b704ac HTTP/1.1" 502 150 2025-12-11 19:53:30,198 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b05ebfee8383e92b704ac'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:53:31,658 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b04d2fee8383e92b6f99c 2025-12-11 19:53:31,660 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b04d2fee8383e92b6f99c HTTP/1.1" 502 150 2025-12-11 19:53:31,660 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b04d2fee8383e92b6f99c'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:47688 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:53:39,660 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b04d2fee8383e92b6f99c 2025-12-11 19:53:39,661 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b04d2fee8383e92b6f99c HTTP/1.1" 502 150 2025-12-11 19:53:39,661 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b04d2fee8383e92b6f99c'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:53:45,242 - lava_callback - INFO - Received callback for node 693af40afee8383e92b6ac06 INFO: 172.21.0.1:52846 - "POST /node/693af40afee8383e92b6ac06 HTTP/1.0" 202 Accepted 2025-12-11 19:53:45,248 - lava_callback - INFO - Starting async job submit for node 693af40afee8383e92b6ac06 2025-12-11 19:53:45,268 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:53:45,277 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af40afee8383e92b6ac06 HTTP/1.1" 502 150 2025-12-11 19:53:45,277 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af40afee8383e92b6ac06'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:53:45,277 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af40afee8383e92b6ac06 2025-12-11 19:53:45,278 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af40afee8383e92b6ac06 HTTP/1.1" 502 150 2025-12-11 19:53:45,278 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af40afee8383e92b6ac06'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:53:45,634 - lava_callback - INFO - Received callback for node 693afcf1fee8383e92b6db7f INFO: 172.21.0.1:52848 - "POST /node/693afcf1fee8383e92b6db7f HTTP/1.0" 202 Accepted 2025-12-11 19:53:45,641 - lava_callback - INFO - Starting async job submit for node 693afcf1fee8383e92b6db7f 2025-12-11 19:53:45,695 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:53:45,705 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afcf1fee8383e92b6db7f HTTP/1.1" 502 150 2025-12-11 19:53:45,705 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afcf1fee8383e92b6db7f'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:53:45,705 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afcf1fee8383e92b6db7f 2025-12-11 19:53:45,706 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afcf1fee8383e92b6db7f HTTP/1.1" 502 150 2025-12-11 19:53:45,706 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afcf1fee8383e92b6db7f'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:53:46,198 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b05ebfee8383e92b704ac 2025-12-11 19:53:46,199 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b05ebfee8383e92b704ac HTTP/1.1" 502 150 2025-12-11 19:53:46,199 - lava_callback - ERROR - Error processing callback for node 693b05ebfee8383e92b704ac: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b05ebfee8383e92b704ac (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b05ebfee8383e92b704ac (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b05ebfee8383e92b704ac (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:52864 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:53:47,278 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af40afee8383e92b6ac06 2025-12-11 19:53:47,279 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af40afee8383e92b6ac06 HTTP/1.1" 502 150 2025-12-11 19:53:47,280 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af40afee8383e92b6ac06'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:53:47,706 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afcf1fee8383e92b6db7f 2025-12-11 19:53:47,710 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afcf1fee8383e92b6db7f HTTP/1.1" 502 150 2025-12-11 19:53:47,710 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afcf1fee8383e92b6db7f'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:53:51,280 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af40afee8383e92b6ac06 2025-12-11 19:53:51,281 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af40afee8383e92b6ac06 HTTP/1.1" 502 150 2025-12-11 19:53:51,281 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af40afee8383e92b6ac06'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:53:51,710 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afcf1fee8383e92b6db7f 2025-12-11 19:53:51,711 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afcf1fee8383e92b6db7f HTTP/1.1" 502 150 2025-12-11 19:53:51,711 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afcf1fee8383e92b6db7f'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:53:55,662 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b04d2fee8383e92b6f99c 2025-12-11 19:53:55,663 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b04d2fee8383e92b6f99c HTTP/1.1" 502 150 2025-12-11 19:53:55,663 - lava_callback - ERROR - Error processing callback for node 693b04d2fee8383e92b6f99c: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b04d2fee8383e92b6f99c (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b04d2fee8383e92b6f99c (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b04d2fee8383e92b6f99c (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:55752 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:53:59,268 - lava_callback - INFO - Received callback for node 693ae596fee8383e92b64937 INFO: 172.21.0.1:55768 - "POST /node/693ae596fee8383e92b64937 HTTP/1.0" 202 Accepted 2025-12-11 19:53:59,275 - lava_callback - INFO - Starting async job submit for node 693ae596fee8383e92b64937 2025-12-11 19:53:59,286 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af40afee8383e92b6ac06 2025-12-11 19:53:59,297 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af40afee8383e92b6ac06 HTTP/1.1" 502 150 2025-12-11 19:53:59,297 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af40afee8383e92b6ac06'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:53:59,297 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:53:59,318 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae596fee8383e92b64937 HTTP/1.1" 502 150 2025-12-11 19:53:59,318 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae596fee8383e92b64937'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:53:59,318 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae596fee8383e92b64937 2025-12-11 19:53:59,319 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae596fee8383e92b64937 HTTP/1.1" 502 150 2025-12-11 19:53:59,319 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae596fee8383e92b64937'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:53:59,712 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afcf1fee8383e92b6db7f 2025-12-11 19:53:59,713 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afcf1fee8383e92b6db7f HTTP/1.1" 502 150 2025-12-11 19:53:59,713 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afcf1fee8383e92b6db7f'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:01,319 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae596fee8383e92b64937 2025-12-11 19:54:01,320 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae596fee8383e92b64937 HTTP/1.1" 502 150 2025-12-11 19:54:01,320 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae596fee8383e92b64937'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:05,202 - lava_callback - INFO - Received callback for node 693b1476fee8383e92b743e8 INFO: 172.21.0.1:53078 - "POST /node/693b1476fee8383e92b743e8 HTTP/1.0" 202 Accepted 2025-12-11 19:54:05,208 - lava_callback - INFO - Starting async job submit for node 693b1476fee8383e92b743e8 2025-12-11 19:54:05,243 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:54:05,254 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1476fee8383e92b743e8 HTTP/1.1" 502 150 2025-12-11 19:54:05,254 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1476fee8383e92b743e8'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:05,254 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1476fee8383e92b743e8 2025-12-11 19:54:05,254 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1476fee8383e92b743e8 HTTP/1.1" 502 150 2025-12-11 19:54:05,255 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1476fee8383e92b743e8'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:05,321 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae596fee8383e92b64937 2025-12-11 19:54:05,322 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae596fee8383e92b64937 HTTP/1.1" 502 150 2025-12-11 19:54:05,322 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae596fee8383e92b64937'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:53092 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:54:07,255 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1476fee8383e92b743e8 2025-12-11 19:54:07,256 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1476fee8383e92b743e8 HTTP/1.1" 502 150 2025-12-11 19:54:07,256 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1476fee8383e92b743e8'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:11,051 - lava_callback - INFO - Received callback for node 693b04f3fee8383e92b6fa08 INFO: 172.21.0.1:43470 - "POST /node/693b04f3fee8383e92b6fa08 HTTP/1.0" 202 Accepted 2025-12-11 19:54:11,060 - lava_callback - INFO - Starting async job submit for node 693b04f3fee8383e92b6fa08 2025-12-11 19:54:11,134 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:54:11,145 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b04f3fee8383e92b6fa08 HTTP/1.1" 502 150 2025-12-11 19:54:11,145 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b04f3fee8383e92b6fa08'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:11,145 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b04f3fee8383e92b6fa08 2025-12-11 19:54:11,146 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b04f3fee8383e92b6fa08 HTTP/1.1" 502 150 2025-12-11 19:54:11,146 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b04f3fee8383e92b6fa08'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:11,256 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1476fee8383e92b743e8 2025-12-11 19:54:11,258 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1476fee8383e92b743e8 HTTP/1.1" 502 150 2025-12-11 19:54:11,259 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1476fee8383e92b743e8'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:13,146 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b04f3fee8383e92b6fa08 2025-12-11 19:54:13,148 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b04f3fee8383e92b6fa08 HTTP/1.1" 502 150 2025-12-11 19:54:13,148 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b04f3fee8383e92b6fa08'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:13,322 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae596fee8383e92b64937 2025-12-11 19:54:13,323 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae596fee8383e92b64937 HTTP/1.1" 502 150 2025-12-11 19:54:13,323 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae596fee8383e92b64937'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:43480 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 19:54:15,297 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af40afee8383e92b6ac06 2025-12-11 19:54:15,298 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af40afee8383e92b6ac06 HTTP/1.1" 502 150 2025-12-11 19:54:15,298 - lava_callback - ERROR - Error processing callback for node 693af40afee8383e92b6ac06: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af40afee8383e92b6ac06 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af40afee8383e92b6ac06 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af40afee8383e92b6ac06 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:54:15,713 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afcf1fee8383e92b6db7f 2025-12-11 19:54:15,714 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afcf1fee8383e92b6db7f HTTP/1.1" 502 150 2025-12-11 19:54:15,715 - lava_callback - ERROR - Error processing callback for node 693afcf1fee8383e92b6db7f: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afcf1fee8383e92b6db7f (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afcf1fee8383e92b6db7f (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afcf1fee8383e92b6db7f (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:43494 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:54:17,148 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b04f3fee8383e92b6fa08 2025-12-11 19:54:17,149 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b04f3fee8383e92b6fa08 HTTP/1.1" 502 150 2025-12-11 19:54:17,149 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b04f3fee8383e92b6fa08'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:17,186 - lava_callback - INFO - Received callback for node 693b14a9fee8383e92b74486 INFO: 172.21.0.1:43504 - "POST /node/693b14a9fee8383e92b74486 HTTP/1.0" 202 Accepted 2025-12-11 19:54:17,192 - lava_callback - INFO - Starting async job submit for node 693b14a9fee8383e92b74486 2025-12-11 19:54:17,214 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:54:17,223 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b14a9fee8383e92b74486 HTTP/1.1" 502 150 2025-12-11 19:54:17,224 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b14a9fee8383e92b74486'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:17,224 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b14a9fee8383e92b74486 2025-12-11 19:54:17,224 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b14a9fee8383e92b74486 HTTP/1.1" 502 150 2025-12-11 19:54:17,224 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b14a9fee8383e92b74486'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:17,817 - lava_callback - INFO - Received callback for node 693b1a8d15f407b7629dfcd6 INFO: 172.21.0.1:43512 - "POST /node/693b1a8d15f407b7629dfcd6 HTTP/1.0" 202 Accepted 2025-12-11 19:54:17,824 - lava_callback - INFO - Starting async job submit for node 693b1a8d15f407b7629dfcd6 2025-12-11 19:54:17,846 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:54:17,856 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a8d15f407b7629dfcd6 HTTP/1.1" 502 150 2025-12-11 19:54:17,856 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a8d15f407b7629dfcd6'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:17,856 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a8d15f407b7629dfcd6 2025-12-11 19:54:17,857 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a8d15f407b7629dfcd6 HTTP/1.1" 502 150 2025-12-11 19:54:17,857 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a8d15f407b7629dfcd6'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:19,225 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b14a9fee8383e92b74486 2025-12-11 19:54:19,226 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b14a9fee8383e92b74486 HTTP/1.1" 502 150 2025-12-11 19:54:19,226 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b14a9fee8383e92b74486'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:19,259 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1476fee8383e92b743e8 2025-12-11 19:54:19,260 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1476fee8383e92b743e8 HTTP/1.1" 502 150 2025-12-11 19:54:19,260 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1476fee8383e92b743e8'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:19,857 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a8d15f407b7629dfcd6 2025-12-11 19:54:19,858 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a8d15f407b7629dfcd6 HTTP/1.1" 502 150 2025-12-11 19:54:19,858 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a8d15f407b7629dfcd6'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:23,226 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b14a9fee8383e92b74486 2025-12-11 19:54:23,227 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b14a9fee8383e92b74486 HTTP/1.1" 502 150 2025-12-11 19:54:23,228 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b14a9fee8383e92b74486'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:23,859 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a8d15f407b7629dfcd6 2025-12-11 19:54:23,860 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a8d15f407b7629dfcd6 HTTP/1.1" 502 150 2025-12-11 19:54:23,860 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a8d15f407b7629dfcd6'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:25,150 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b04f3fee8383e92b6fa08 2025-12-11 19:54:25,151 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b04f3fee8383e92b6fa08 HTTP/1.1" 502 150 2025-12-11 19:54:25,151 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b04f3fee8383e92b6fa08'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:40618 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40634 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:54:28,511 - lava_callback - INFO - Received callback for node 693b06f9fee8383e92b70966 INFO: 172.21.0.1:40638 - "POST /node/693b06f9fee8383e92b70966 HTTP/1.0" 202 Accepted 2025-12-11 19:54:28,520 - lava_callback - INFO - Starting async job submit for node 693b06f9fee8383e92b70966 2025-12-11 19:54:28,595 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:54:28,605 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b06f9fee8383e92b70966 HTTP/1.1" 502 150 2025-12-11 19:54:28,605 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b06f9fee8383e92b70966'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:28,605 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b06f9fee8383e92b70966 2025-12-11 19:54:28,606 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b06f9fee8383e92b70966 HTTP/1.1" 502 150 2025-12-11 19:54:28,606 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b06f9fee8383e92b70966'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:29,324 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae596fee8383e92b64937 2025-12-11 19:54:29,325 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae596fee8383e92b64937 HTTP/1.1" 502 150 2025-12-11 19:54:29,325 - lava_callback - ERROR - Error processing callback for node 693ae596fee8383e92b64937: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae596fee8383e92b64937 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae596fee8383e92b64937 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae596fee8383e92b64937 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:54:30,606 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b06f9fee8383e92b70966 2025-12-11 19:54:30,607 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b06f9fee8383e92b70966 HTTP/1.1" 502 150 2025-12-11 19:54:30,607 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b06f9fee8383e92b70966'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:31,228 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b14a9fee8383e92b74486 2025-12-11 19:54:31,229 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b14a9fee8383e92b74486 HTTP/1.1" 502 150 2025-12-11 19:54:31,229 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b14a9fee8383e92b74486'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:31,860 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a8d15f407b7629dfcd6 2025-12-11 19:54:31,861 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a8d15f407b7629dfcd6 HTTP/1.1" 502 150 2025-12-11 19:54:31,862 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a8d15f407b7629dfcd6'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:34,608 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b06f9fee8383e92b70966 2025-12-11 19:54:34,609 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b06f9fee8383e92b70966 HTTP/1.1" 502 150 2025-12-11 19:54:34,609 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b06f9fee8383e92b70966'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:35,260 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1476fee8383e92b743e8 2025-12-11 19:54:35,261 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1476fee8383e92b743e8 HTTP/1.1" 502 150 2025-12-11 19:54:35,262 - lava_callback - ERROR - Error processing callback for node 693b1476fee8383e92b743e8: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1476fee8383e92b743e8 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1476fee8383e92b743e8 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1476fee8383e92b743e8 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:49640 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:54:41,151 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b04f3fee8383e92b6fa08 2025-12-11 19:54:41,153 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b04f3fee8383e92b6fa08 HTTP/1.1" 502 150 2025-12-11 19:54:41,153 - lava_callback - ERROR - Error processing callback for node 693b04f3fee8383e92b6fa08: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b04f3fee8383e92b6fa08 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b04f3fee8383e92b6fa08 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b04f3fee8383e92b6fa08 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:54:42,609 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b06f9fee8383e92b70966 2025-12-11 19:54:42,610 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b06f9fee8383e92b70966 HTTP/1.1" 502 150 2025-12-11 19:54:42,610 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b06f9fee8383e92b70966'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:45,669 - lava_callback - INFO - Received callback for node 693b14a3fee8383e92b74473 INFO: 172.21.0.1:33318 - "POST /node/693b14a3fee8383e92b74473 HTTP/1.0" 202 Accepted 2025-12-11 19:54:45,676 - lava_callback - INFO - Starting async job submit for node 693b14a3fee8383e92b74473 2025-12-11 19:54:45,727 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:54:45,740 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b14a3fee8383e92b74473 HTTP/1.1" 502 150 2025-12-11 19:54:45,740 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b14a3fee8383e92b74473'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:45,740 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b14a3fee8383e92b74473 2025-12-11 19:54:45,741 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b14a3fee8383e92b74473 HTTP/1.1" 502 150 2025-12-11 19:54:45,741 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b14a3fee8383e92b74473'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:33324 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:54:47,229 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b14a9fee8383e92b74486 2025-12-11 19:54:47,231 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b14a9fee8383e92b74486 HTTP/1.1" 502 150 2025-12-11 19:54:47,231 - lava_callback - ERROR - Error processing callback for node 693b14a9fee8383e92b74486: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b14a9fee8383e92b74486 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b14a9fee8383e92b74486 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b14a9fee8383e92b74486 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:54:47,742 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b14a3fee8383e92b74473 2025-12-11 19:54:47,743 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b14a3fee8383e92b74473 HTTP/1.1" 502 150 2025-12-11 19:54:47,743 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b14a3fee8383e92b74473'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:47,862 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a8d15f407b7629dfcd6 2025-12-11 19:54:47,863 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a8d15f407b7629dfcd6 HTTP/1.1" 502 150 2025-12-11 19:54:47,863 - lava_callback - ERROR - Error processing callback for node 693b1a8d15f407b7629dfcd6: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1a8d15f407b7629dfcd6 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1a8d15f407b7629dfcd6 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1a8d15f407b7629dfcd6 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:54:51,743 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b14a3fee8383e92b74473 2025-12-11 19:54:51,744 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b14a3fee8383e92b74473 HTTP/1.1" 502 150 2025-12-11 19:54:51,744 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b14a3fee8383e92b74473'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:48172 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:54:58,417 - lava_callback - INFO - Received callback for node 693b1a9315f407b7629dfce8 INFO: 172.21.0.1:48188 - "POST /node/693b1a9315f407b7629dfce8 HTTP/1.0" 202 Accepted 2025-12-11 19:54:58,428 - lava_callback - INFO - Starting async job submit for node 693b1a9315f407b7629dfce8 2025-12-11 19:54:58,466 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:54:58,485 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a9315f407b7629dfce8 HTTP/1.1" 502 150 2025-12-11 19:54:58,486 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a9315f407b7629dfce8'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:58,486 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a9315f407b7629dfce8 2025-12-11 19:54:58,486 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a9315f407b7629dfce8 HTTP/1.1" 502 150 2025-12-11 19:54:58,487 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a9315f407b7629dfce8'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:54:58,610 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b06f9fee8383e92b70966 2025-12-11 19:54:58,612 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b06f9fee8383e92b70966 HTTP/1.1" 502 150 2025-12-11 19:54:58,612 - lava_callback - ERROR - Error processing callback for node 693b06f9fee8383e92b70966: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b06f9fee8383e92b70966 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b06f9fee8383e92b70966 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b06f9fee8383e92b70966 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:54:59,745 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b14a3fee8383e92b74473 2025-12-11 19:54:59,746 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b14a3fee8383e92b74473 HTTP/1.1" 502 150 2025-12-11 19:54:59,746 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b14a3fee8383e92b74473'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:55:00,487 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a9315f407b7629dfce8 2025-12-11 19:55:00,488 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a9315f407b7629dfce8 HTTP/1.1" 502 150 2025-12-11 19:55:00,488 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a9315f407b7629dfce8'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:55:04,488 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a9315f407b7629dfce8 2025-12-11 19:55:04,490 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a9315f407b7629dfce8 HTTP/1.1" 502 150 2025-12-11 19:55:04,490 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a9315f407b7629dfce8'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:52626 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:55:12,490 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a9315f407b7629dfce8 2025-12-11 19:55:12,491 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a9315f407b7629dfce8 HTTP/1.1" 502 150 2025-12-11 19:55:12,491 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a9315f407b7629dfce8'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:39678 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 19:55:15,746 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b14a3fee8383e92b74473 2025-12-11 19:55:15,747 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b14a3fee8383e92b74473 HTTP/1.1" 502 150 2025-12-11 19:55:15,747 - lava_callback - ERROR - Error processing callback for node 693b14a3fee8383e92b74473: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b14a3fee8383e92b74473 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b14a3fee8383e92b74473 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b14a3fee8383e92b74473 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:39684 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47932 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47944 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:55:28,491 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a9315f407b7629dfce8 2025-12-11 19:55:28,492 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a9315f407b7629dfce8 HTTP/1.1" 502 150 2025-12-11 19:55:28,493 - lava_callback - ERROR - Error processing callback for node 693b1a9315f407b7629dfce8: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1a9315f407b7629dfce8 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1a9315f407b7629dfce8 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1a9315f407b7629dfce8 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:55:33,681 - lava_callback - INFO - Received callback for node 693b07c1fee8383e92b71011 INFO: 172.21.0.1:32962 - "POST /node/693b07c1fee8383e92b71011 HTTP/1.0" 202 Accepted 2025-12-11 19:55:33,689 - lava_callback - INFO - Starting async job submit for node 693b07c1fee8383e92b71011 2025-12-11 19:55:33,732 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:55:33,767 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07c1fee8383e92b71011 HTTP/1.1" 502 150 2025-12-11 19:55:33,767 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07c1fee8383e92b71011'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:55:33,767 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07c1fee8383e92b71011 2025-12-11 19:55:33,768 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07c1fee8383e92b71011 HTTP/1.1" 502 150 2025-12-11 19:55:33,768 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07c1fee8383e92b71011'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:55:35,768 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07c1fee8383e92b71011 2025-12-11 19:55:35,769 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07c1fee8383e92b71011 HTTP/1.1" 502 150 2025-12-11 19:55:35,770 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07c1fee8383e92b71011'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:32966 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:55:39,770 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07c1fee8383e92b71011 2025-12-11 19:55:39,771 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07c1fee8383e92b71011 HTTP/1.1" 502 150 2025-12-11 19:55:39,771 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07c1fee8383e92b71011'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:55:46,236 - lava_callback - INFO - Received callback for node 693ae854fee8383e92b658ad INFO: 172.21.0.1:38012 - "POST /node/693ae854fee8383e92b658ad HTTP/1.0" 202 Accepted 2025-12-11 19:55:46,243 - lava_callback - INFO - Starting async job submit for node 693ae854fee8383e92b658ad 2025-12-11 19:55:46,268 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:55:46,278 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae854fee8383e92b658ad HTTP/1.1" 502 150 2025-12-11 19:55:46,278 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae854fee8383e92b658ad'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:55:46,279 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae854fee8383e92b658ad 2025-12-11 19:55:46,279 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae854fee8383e92b658ad HTTP/1.1" 502 150 2025-12-11 19:55:46,279 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae854fee8383e92b658ad'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:38028 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:55:47,771 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07c1fee8383e92b71011 2025-12-11 19:55:47,772 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07c1fee8383e92b71011 HTTP/1.1" 502 150 2025-12-11 19:55:47,773 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07c1fee8383e92b71011'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:55:48,280 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae854fee8383e92b658ad 2025-12-11 19:55:48,281 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae854fee8383e92b658ad HTTP/1.1" 502 150 2025-12-11 19:55:48,281 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae854fee8383e92b658ad'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:55:50,348 - lava_callback - INFO - Received callback for node 693ae6f6fee8383e92b64f5f INFO: 172.21.0.1:54898 - "POST /node/693ae6f6fee8383e92b64f5f HTTP/1.0" 202 Accepted 2025-12-11 19:55:50,375 - lava_callback - INFO - Starting async job submit for node 693ae6f6fee8383e92b64f5f 2025-12-11 19:55:50,405 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:55:50,415 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae6f6fee8383e92b64f5f HTTP/1.1" 502 150 2025-12-11 19:55:50,415 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae6f6fee8383e92b64f5f'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:55:50,415 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae6f6fee8383e92b64f5f 2025-12-11 19:55:50,416 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae6f6fee8383e92b64f5f HTTP/1.1" 502 150 2025-12-11 19:55:50,416 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae6f6fee8383e92b64f5f'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:55:52,281 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae854fee8383e92b658ad 2025-12-11 19:55:52,283 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae854fee8383e92b658ad HTTP/1.1" 502 150 2025-12-11 19:55:52,283 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae854fee8383e92b658ad'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:55:52,416 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae6f6fee8383e92b64f5f 2025-12-11 19:55:52,417 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae6f6fee8383e92b64f5f HTTP/1.1" 502 150 2025-12-11 19:55:52,417 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae6f6fee8383e92b64f5f'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:55:56,418 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae6f6fee8383e92b64f5f 2025-12-11 19:55:56,419 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae6f6fee8383e92b64f5f HTTP/1.1" 502 150 2025-12-11 19:55:56,419 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae6f6fee8383e92b64f5f'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:54904 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:56:00,283 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae854fee8383e92b658ad 2025-12-11 19:56:00,285 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae854fee8383e92b658ad HTTP/1.1" 502 150 2025-12-11 19:56:00,285 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae854fee8383e92b658ad'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:03,773 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07c1fee8383e92b71011 2025-12-11 19:56:03,774 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07c1fee8383e92b71011 HTTP/1.1" 502 150 2025-12-11 19:56:03,774 - lava_callback - ERROR - Error processing callback for node 693b07c1fee8383e92b71011: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b07c1fee8383e92b71011 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b07c1fee8383e92b71011 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b07c1fee8383e92b71011 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:56:04,419 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae6f6fee8383e92b64f5f 2025-12-11 19:56:04,420 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae6f6fee8383e92b64f5f HTTP/1.1" 502 150 2025-12-11 19:56:04,420 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae6f6fee8383e92b64f5f'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:05,268 - lava_callback - INFO - Received callback for node 693af46afee8383e92b6ae36 INFO: 172.21.0.1:55290 - "POST /node/693af46afee8383e92b6ae36 HTTP/1.0" 202 Accepted 2025-12-11 19:56:05,275 - lava_callback - INFO - Starting async job submit for node 693af46afee8383e92b6ae36 2025-12-11 19:56:05,293 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:56:05,304 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af46afee8383e92b6ae36 HTTP/1.1" 502 150 2025-12-11 19:56:05,304 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af46afee8383e92b6ae36'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:05,304 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af46afee8383e92b6ae36 2025-12-11 19:56:05,304 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af46afee8383e92b6ae36 HTTP/1.1" 502 150 2025-12-11 19:56:05,305 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af46afee8383e92b6ae36'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:06,201 - lava_callback - INFO - Received callback for node 693ae857fee8383e92b658bf INFO: 172.21.0.1:55296 - "POST /node/693ae857fee8383e92b658bf HTTP/1.0" 202 Accepted 2025-12-11 19:56:06,208 - lava_callback - INFO - Starting async job submit for node 693ae857fee8383e92b658bf 2025-12-11 19:56:06,238 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:56:06,248 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae857fee8383e92b658bf HTTP/1.1" 502 150 2025-12-11 19:56:06,248 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae857fee8383e92b658bf'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:06,248 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae857fee8383e92b658bf 2025-12-11 19:56:06,249 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae857fee8383e92b658bf HTTP/1.1" 502 150 2025-12-11 19:56:06,249 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae857fee8383e92b658bf'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:55300 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:56:07,305 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af46afee8383e92b6ae36 2025-12-11 19:56:07,306 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af46afee8383e92b6ae36 HTTP/1.1" 502 150 2025-12-11 19:56:07,306 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af46afee8383e92b6ae36'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:08,249 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae857fee8383e92b658bf 2025-12-11 19:56:08,250 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae857fee8383e92b658bf HTTP/1.1" 502 150 2025-12-11 19:56:08,250 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae857fee8383e92b658bf'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:09,990 - lava_callback - INFO - Received callback for node 693b1aba15f407b7629dfd89 INFO: 172.21.0.1:56576 - "POST /node/693b1aba15f407b7629dfd89 HTTP/1.0" 202 Accepted 2025-12-11 19:56:09,997 - lava_callback - INFO - Starting async job submit for node 693b1aba15f407b7629dfd89 2025-12-11 19:56:10,020 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:56:10,030 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1aba15f407b7629dfd89 HTTP/1.1" 502 150 2025-12-11 19:56:10,030 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1aba15f407b7629dfd89'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:10,030 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1aba15f407b7629dfd89 2025-12-11 19:56:10,031 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1aba15f407b7629dfd89 HTTP/1.1" 502 150 2025-12-11 19:56:10,031 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1aba15f407b7629dfd89'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:11,306 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af46afee8383e92b6ae36 2025-12-11 19:56:11,308 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af46afee8383e92b6ae36 HTTP/1.1" 502 150 2025-12-11 19:56:11,308 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af46afee8383e92b6ae36'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:11,393 - lava_callback - INFO - Received callback for node 693b05ebfee8383e92b704af INFO: 172.21.0.1:56578 - "POST /node/693b05ebfee8383e92b704af HTTP/1.0" 202 Accepted 2025-12-11 19:56:11,401 - lava_callback - INFO - Starting async job submit for node 693b05ebfee8383e92b704af 2025-12-11 19:56:11,421 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:56:11,434 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b05ebfee8383e92b704af HTTP/1.1" 502 150 2025-12-11 19:56:11,434 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b05ebfee8383e92b704af'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:11,434 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b05ebfee8383e92b704af 2025-12-11 19:56:11,435 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b05ebfee8383e92b704af HTTP/1.1" 502 150 2025-12-11 19:56:11,435 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b05ebfee8383e92b704af'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:12,031 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1aba15f407b7629dfd89 2025-12-11 19:56:12,032 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1aba15f407b7629dfd89 HTTP/1.1" 502 150 2025-12-11 19:56:12,033 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1aba15f407b7629dfd89'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:12,250 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae857fee8383e92b658bf 2025-12-11 19:56:12,252 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae857fee8383e92b658bf HTTP/1.1" 502 150 2025-12-11 19:56:12,252 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae857fee8383e92b658bf'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:13,436 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b05ebfee8383e92b704af 2025-12-11 19:56:13,437 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b05ebfee8383e92b704af HTTP/1.1" 502 150 2025-12-11 19:56:13,437 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b05ebfee8383e92b704af'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:56588 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 19:56:15,694 - lava_callback - INFO - Received callback for node 693b1ab415f407b7629dfd7c INFO: 172.21.0.1:56592 - "POST /node/693b1ab415f407b7629dfd7c HTTP/1.0" 202 Accepted 2025-12-11 19:56:15,701 - lava_callback - INFO - Starting async job submit for node 693b1ab415f407b7629dfd7c 2025-12-11 19:56:15,724 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:56:15,734 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ab415f407b7629dfd7c HTTP/1.1" 502 150 2025-12-11 19:56:15,734 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ab415f407b7629dfd7c'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:15,734 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ab415f407b7629dfd7c 2025-12-11 19:56:15,735 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ab415f407b7629dfd7c HTTP/1.1" 502 150 2025-12-11 19:56:15,735 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ab415f407b7629dfd7c'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:16,033 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1aba15f407b7629dfd89 2025-12-11 19:56:16,034 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1aba15f407b7629dfd89 HTTP/1.1" 502 150 2025-12-11 19:56:16,034 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1aba15f407b7629dfd89'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:16,285 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae854fee8383e92b658ad 2025-12-11 19:56:16,286 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae854fee8383e92b658ad HTTP/1.1" 502 150 2025-12-11 19:56:16,286 - lava_callback - ERROR - Error processing callback for node 693ae854fee8383e92b658ad: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae854fee8383e92b658ad (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae854fee8383e92b658ad (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae854fee8383e92b658ad (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:56602 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:56:17,437 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b05ebfee8383e92b704af 2025-12-11 19:56:17,439 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b05ebfee8383e92b704af HTTP/1.1" 502 150 2025-12-11 19:56:17,439 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b05ebfee8383e92b704af'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:17,735 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ab415f407b7629dfd7c 2025-12-11 19:56:17,737 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ab415f407b7629dfd7c HTTP/1.1" 502 150 2025-12-11 19:56:17,737 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ab415f407b7629dfd7c'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:19,308 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af46afee8383e92b6ae36 2025-12-11 19:56:19,309 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af46afee8383e92b6ae36 HTTP/1.1" 502 150 2025-12-11 19:56:19,309 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af46afee8383e92b6ae36'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:20,252 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae857fee8383e92b658bf 2025-12-11 19:56:20,253 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae857fee8383e92b658bf HTTP/1.1" 502 150 2025-12-11 19:56:20,253 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae857fee8383e92b658bf'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:20,420 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae6f6fee8383e92b64f5f 2025-12-11 19:56:20,422 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae6f6fee8383e92b64f5f HTTP/1.1" 502 150 2025-12-11 19:56:20,422 - lava_callback - ERROR - Error processing callback for node 693ae6f6fee8383e92b64f5f: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae6f6fee8383e92b64f5f (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae6f6fee8383e92b64f5f (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae6f6fee8383e92b64f5f (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:56:21,737 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ab415f407b7629dfd7c 2025-12-11 19:56:21,738 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ab415f407b7629dfd7c HTTP/1.1" 502 150 2025-12-11 19:56:21,738 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ab415f407b7629dfd7c'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:24,034 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1aba15f407b7629dfd89 2025-12-11 19:56:24,036 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1aba15f407b7629dfd89 HTTP/1.1" 502 150 2025-12-11 19:56:24,036 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1aba15f407b7629dfd89'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:25,439 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b05ebfee8383e92b704af 2025-12-11 19:56:25,440 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b05ebfee8383e92b704af HTTP/1.1" 502 150 2025-12-11 19:56:25,440 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b05ebfee8383e92b704af'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:46688 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46692 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:56:29,739 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ab415f407b7629dfd7c 2025-12-11 19:56:29,740 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ab415f407b7629dfd7c HTTP/1.1" 502 150 2025-12-11 19:56:29,740 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ab415f407b7629dfd7c'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:35,310 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af46afee8383e92b6ae36 2025-12-11 19:56:35,311 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af46afee8383e92b6ae36 HTTP/1.1" 502 150 2025-12-11 19:56:35,311 - lava_callback - ERROR - Error processing callback for node 693af46afee8383e92b6ae36: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af46afee8383e92b6ae36 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af46afee8383e92b6ae36 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af46afee8383e92b6ae36 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:56:36,253 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae857fee8383e92b658bf 2025-12-11 19:56:36,254 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae857fee8383e92b658bf HTTP/1.1" 502 150 2025-12-11 19:56:36,255 - lava_callback - ERROR - Error processing callback for node 693ae857fee8383e92b658bf: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae857fee8383e92b658bf (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae857fee8383e92b658bf (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae857fee8383e92b658bf (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:43236 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:56:40,036 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1aba15f407b7629dfd89 2025-12-11 19:56:40,038 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1aba15f407b7629dfd89 HTTP/1.1" 502 150 2025-12-11 19:56:40,038 - lava_callback - ERROR - Error processing callback for node 693b1aba15f407b7629dfd89: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1aba15f407b7629dfd89 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1aba15f407b7629dfd89 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1aba15f407b7629dfd89 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:56:41,441 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b05ebfee8383e92b704af 2025-12-11 19:56:41,443 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b05ebfee8383e92b704af HTTP/1.1" 502 150 2025-12-11 19:56:41,443 - lava_callback - ERROR - Error processing callback for node 693b05ebfee8383e92b704af: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b05ebfee8383e92b704af (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b05ebfee8383e92b704af (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b05ebfee8383e92b704af (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:56:45,508 - lava_callback - INFO - Received callback for node 693b0a21fee8383e92b719a1 INFO: 172.21.0.1:45900 - "POST /node/693b0a21fee8383e92b719a1 HTTP/1.0" 202 Accepted 2025-12-11 19:56:45,515 - lava_callback - INFO - Starting async job submit for node 693b0a21fee8383e92b719a1 2025-12-11 19:56:45,577 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:56:45,597 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0a21fee8383e92b719a1 HTTP/1.1" 502 150 2025-12-11 19:56:45,597 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0a21fee8383e92b719a1'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:45,597 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0a21fee8383e92b719a1 2025-12-11 19:56:45,598 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0a21fee8383e92b719a1 HTTP/1.1" 502 150 2025-12-11 19:56:45,598 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0a21fee8383e92b719a1'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:45,740 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ab415f407b7629dfd7c 2025-12-11 19:56:45,741 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ab415f407b7629dfd7c HTTP/1.1" 502 150 2025-12-11 19:56:45,741 - lava_callback - ERROR - Error processing callback for node 693b1ab415f407b7629dfd7c: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1ab415f407b7629dfd7c (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1ab415f407b7629dfd7c (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1ab415f407b7629dfd7c (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:45912 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:56:47,176 - lava_callback - INFO - Received callback for node 693b123efee8383e92b73c01 INFO: 172.21.0.1:45926 - "POST /node/693b123efee8383e92b73c01 HTTP/1.0" 202 Accepted 2025-12-11 19:56:47,183 - lava_callback - INFO - Starting async job submit for node 693b123efee8383e92b73c01 2025-12-11 19:56:47,221 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:56:47,256 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b123efee8383e92b73c01 HTTP/1.1" 502 150 2025-12-11 19:56:47,256 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b123efee8383e92b73c01'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:47,256 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b123efee8383e92b73c01 2025-12-11 19:56:47,257 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b123efee8383e92b73c01 HTTP/1.1" 502 150 2025-12-11 19:56:47,257 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b123efee8383e92b73c01'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:47,598 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0a21fee8383e92b719a1 2025-12-11 19:56:47,599 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0a21fee8383e92b719a1 HTTP/1.1" 502 150 2025-12-11 19:56:47,599 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0a21fee8383e92b719a1'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:49,050 - lava_callback - INFO - Received callback for node 693b082ffee8383e92b71203 INFO: 172.21.0.1:45928 - "POST /node/693b082ffee8383e92b71203 HTTP/1.0" 202 Accepted 2025-12-11 19:56:49,058 - lava_callback - INFO - Starting async job submit for node 693b082ffee8383e92b71203 2025-12-11 19:56:49,086 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:56:49,100 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b082ffee8383e92b71203 HTTP/1.1" 502 150 2025-12-11 19:56:49,100 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b082ffee8383e92b71203'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:49,100 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b082ffee8383e92b71203 2025-12-11 19:56:49,101 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b082ffee8383e92b71203 HTTP/1.1" 502 150 2025-12-11 19:56:49,101 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b082ffee8383e92b71203'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:49,257 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b123efee8383e92b73c01 2025-12-11 19:56:49,258 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b123efee8383e92b73c01 HTTP/1.1" 502 150 2025-12-11 19:56:49,258 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b123efee8383e92b73c01'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:49,721 - lava_callback - INFO - Received callback for node 693b071bfee8383e92b70b35 INFO: 172.21.0.1:46846 - "POST /node/693b071bfee8383e92b70b35 HTTP/1.0" 202 Accepted 2025-12-11 19:56:49,730 - lava_callback - INFO - Starting async job submit for node 693b071bfee8383e92b70b35 2025-12-11 19:56:49,792 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:56:49,801 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b071bfee8383e92b70b35 HTTP/1.1" 502 150 2025-12-11 19:56:49,801 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b071bfee8383e92b70b35'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:49,802 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b071bfee8383e92b70b35 2025-12-11 19:56:49,802 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b071bfee8383e92b70b35 HTTP/1.1" 502 150 2025-12-11 19:56:49,802 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b071bfee8383e92b70b35'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:51,101 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b082ffee8383e92b71203 2025-12-11 19:56:51,102 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b082ffee8383e92b71203 HTTP/1.1" 502 150 2025-12-11 19:56:51,102 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b082ffee8383e92b71203'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:51,600 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0a21fee8383e92b719a1 2025-12-11 19:56:51,601 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0a21fee8383e92b719a1 HTTP/1.1" 502 150 2025-12-11 19:56:51,601 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0a21fee8383e92b719a1'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:51,802 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b071bfee8383e92b70b35 2025-12-11 19:56:51,803 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b071bfee8383e92b70b35 HTTP/1.1" 502 150 2025-12-11 19:56:51,803 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b071bfee8383e92b70b35'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:53,258 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b123efee8383e92b73c01 2025-12-11 19:56:53,260 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b123efee8383e92b73c01 HTTP/1.1" 502 150 2025-12-11 19:56:53,260 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b123efee8383e92b73c01'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:55,102 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b082ffee8383e92b71203 2025-12-11 19:56:55,104 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b082ffee8383e92b71203 HTTP/1.1" 502 150 2025-12-11 19:56:55,104 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b082ffee8383e92b71203'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:56:55,804 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b071bfee8383e92b70b35 2025-12-11 19:56:55,805 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b071bfee8383e92b70b35 HTTP/1.1" 502 150 2025-12-11 19:56:55,805 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b071bfee8383e92b70b35'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:46852 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:56:59,601 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0a21fee8383e92b719a1 2025-12-11 19:56:59,602 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0a21fee8383e92b719a1 HTTP/1.1" 502 150 2025-12-11 19:56:59,602 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0a21fee8383e92b719a1'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:57:01,260 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b123efee8383e92b73c01 2025-12-11 19:57:01,261 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b123efee8383e92b73c01 HTTP/1.1" 502 150 2025-12-11 19:57:01,261 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b123efee8383e92b73c01'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:57:03,104 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b082ffee8383e92b71203 2025-12-11 19:57:03,106 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b082ffee8383e92b71203 HTTP/1.1" 502 150 2025-12-11 19:57:03,106 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b082ffee8383e92b71203'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:57:03,805 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b071bfee8383e92b70b35 2025-12-11 19:57:03,808 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b071bfee8383e92b70b35 HTTP/1.1" 502 150 2025-12-11 19:57:03,808 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b071bfee8383e92b70b35'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:40174 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:57:12,526 - lava_callback - INFO - Received callback for node 693b14effee8383e92b7455a INFO: 172.21.0.1:34924 - "POST /node/693b14effee8383e92b7455a HTTP/1.0" 202 Accepted 2025-12-11 19:57:12,533 - lava_callback - INFO - Starting async job submit for node 693b14effee8383e92b7455a 2025-12-11 19:57:12,554 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:57:12,564 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b14effee8383e92b7455a HTTP/1.1" 502 150 2025-12-11 19:57:12,564 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b14effee8383e92b7455a'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:57:12,564 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b14effee8383e92b7455a 2025-12-11 19:57:12,565 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b14effee8383e92b7455a HTTP/1.1" 502 150 2025-12-11 19:57:12,565 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b14effee8383e92b7455a'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:57:14,565 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b14effee8383e92b7455a 2025-12-11 19:57:14,567 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b14effee8383e92b7455a HTTP/1.1" 502 150 2025-12-11 19:57:14,567 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b14effee8383e92b7455a'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:34940 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 19:57:15,602 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0a21fee8383e92b719a1 2025-12-11 19:57:15,603 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0a21fee8383e92b719a1 HTTP/1.1" 502 150 2025-12-11 19:57:15,604 - lava_callback - ERROR - Error processing callback for node 693b0a21fee8383e92b719a1: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0a21fee8383e92b719a1 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0a21fee8383e92b719a1 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0a21fee8383e92b719a1 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:34948 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:57:17,261 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b123efee8383e92b73c01 2025-12-11 19:57:17,263 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b123efee8383e92b73c01 HTTP/1.1" 502 150 2025-12-11 19:57:17,263 - lava_callback - ERROR - Error processing callback for node 693b123efee8383e92b73c01: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b123efee8383e92b73c01 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b123efee8383e92b73c01 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b123efee8383e92b73c01 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:57:18,567 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b14effee8383e92b7455a 2025-12-11 19:57:18,568 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b14effee8383e92b7455a HTTP/1.1" 502 150 2025-12-11 19:57:18,568 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b14effee8383e92b7455a'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:57:19,106 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b082ffee8383e92b71203 2025-12-11 19:57:19,108 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b082ffee8383e92b71203 HTTP/1.1" 502 150 2025-12-11 19:57:19,108 - lava_callback - ERROR - Error processing callback for node 693b082ffee8383e92b71203: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b082ffee8383e92b71203 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b082ffee8383e92b71203 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b082ffee8383e92b71203 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:57:19,808 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b071bfee8383e92b70b35 2025-12-11 19:57:19,809 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b071bfee8383e92b70b35 HTTP/1.1" 502 150 2025-12-11 19:57:19,809 - lava_callback - ERROR - Error processing callback for node 693b071bfee8383e92b70b35: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b071bfee8383e92b70b35 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b071bfee8383e92b70b35 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b071bfee8383e92b70b35 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:57:25,487 - lava_callback - INFO - Received callback for node 693af1d1fee8383e92b6a062 INFO: 172.21.0.1:53402 - "POST /node/693af1d1fee8383e92b6a062 HTTP/1.0" 202 Accepted 2025-12-11 19:57:25,498 - lava_callback - INFO - Starting async job submit for node 693af1d1fee8383e92b6a062 2025-12-11 19:57:25,535 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:57:25,548 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af1d1fee8383e92b6a062 HTTP/1.1" 502 150 2025-12-11 19:57:25,548 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af1d1fee8383e92b6a062'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:57:25,549 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af1d1fee8383e92b6a062 2025-12-11 19:57:25,549 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af1d1fee8383e92b6a062 HTTP/1.1" 502 150 2025-12-11 19:57:25,549 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af1d1fee8383e92b6a062'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:53416 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:57:26,569 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b14effee8383e92b7455a 2025-12-11 19:57:26,570 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b14effee8383e92b7455a HTTP/1.1" 502 150 2025-12-11 19:57:26,570 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b14effee8383e92b7455a'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:53430 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:57:27,550 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af1d1fee8383e92b6a062 2025-12-11 19:57:27,551 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af1d1fee8383e92b6a062 HTTP/1.1" 502 150 2025-12-11 19:57:27,551 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af1d1fee8383e92b6a062'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:57:29,418 - lava_callback - INFO - Received callback for node 693ae6f9fee8383e92b64f7a INFO: 172.21.0.1:53446 - "POST /node/693ae6f9fee8383e92b64f7a HTTP/1.0" 202 Accepted 2025-12-11 19:57:29,424 - lava_callback - INFO - Starting async job submit for node 693ae6f9fee8383e92b64f7a 2025-12-11 19:57:29,455 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:57:29,468 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae6f9fee8383e92b64f7a HTTP/1.1" 502 150 2025-12-11 19:57:29,468 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae6f9fee8383e92b64f7a'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:57:29,468 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae6f9fee8383e92b64f7a 2025-12-11 19:57:29,468 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae6f9fee8383e92b64f7a HTTP/1.1" 502 150 2025-12-11 19:57:29,469 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae6f9fee8383e92b64f7a'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:57:31,469 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae6f9fee8383e92b64f7a 2025-12-11 19:57:31,470 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae6f9fee8383e92b64f7a HTTP/1.1" 502 150 2025-12-11 19:57:31,470 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae6f9fee8383e92b64f7a'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:57:31,551 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af1d1fee8383e92b6a062 2025-12-11 19:57:31,552 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af1d1fee8383e92b6a062 HTTP/1.1" 502 150 2025-12-11 19:57:31,552 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af1d1fee8383e92b6a062'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:57:35,470 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae6f9fee8383e92b64f7a 2025-12-11 19:57:35,472 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae6f9fee8383e92b64f7a HTTP/1.1" 502 150 2025-12-11 19:57:35,472 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae6f9fee8383e92b64f7a'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:44004 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:57:39,552 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af1d1fee8383e92b6a062 2025-12-11 19:57:39,553 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af1d1fee8383e92b6a062 HTTP/1.1" 502 150 2025-12-11 19:57:39,553 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af1d1fee8383e92b6a062'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:57:42,570 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b14effee8383e92b7455a 2025-12-11 19:57:42,572 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b14effee8383e92b7455a HTTP/1.1" 502 150 2025-12-11 19:57:42,572 - lava_callback - ERROR - Error processing callback for node 693b14effee8383e92b7455a: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b14effee8383e92b7455a (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b14effee8383e92b7455a (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b14effee8383e92b7455a (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:57:43,472 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae6f9fee8383e92b64f7a 2025-12-11 19:57:43,473 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae6f9fee8383e92b64f7a HTTP/1.1" 502 150 2025-12-11 19:57:43,473 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae6f9fee8383e92b64f7a'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:50418 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:57:55,554 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af1d1fee8383e92b6a062 2025-12-11 19:57:55,555 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af1d1fee8383e92b6a062 HTTP/1.1" 502 150 2025-12-11 19:57:55,555 - lava_callback - ERROR - Error processing callback for node 693af1d1fee8383e92b6a062: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af1d1fee8383e92b6a062 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af1d1fee8383e92b6a062 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af1d1fee8383e92b6a062 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:53478 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:57:59,473 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae6f9fee8383e92b64f7a 2025-12-11 19:57:59,474 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae6f9fee8383e92b64f7a HTTP/1.1" 502 150 2025-12-11 19:57:59,474 - lava_callback - ERROR - Error processing callback for node 693ae6f9fee8383e92b64f7a: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae6f9fee8383e92b64f7a (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae6f9fee8383e92b64f7a (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae6f9fee8383e92b64f7a (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:43818 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:58:14,030 - lava_callback - INFO - Received callback for node 693b1b6915f407b7629e01c3 INFO: 172.21.0.1:35560 - "POST /node/693b1b6915f407b7629e01c3 HTTP/1.0" 202 Accepted 2025-12-11 19:58:14,041 - lava_callback - INFO - Starting async job submit for node 693b1b6915f407b7629e01c3 2025-12-11 19:58:14,065 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:58:14,082 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1b6915f407b7629e01c3 HTTP/1.1" 502 150 2025-12-11 19:58:14,082 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1b6915f407b7629e01c3'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:58:14,083 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1b6915f407b7629e01c3 2025-12-11 19:58:14,083 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1b6915f407b7629e01c3 HTTP/1.1" 502 150 2025-12-11 19:58:14,083 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1b6915f407b7629e01c3'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:35576 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 19:58:16,084 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1b6915f407b7629e01c3 2025-12-11 19:58:16,085 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1b6915f407b7629e01c3 HTTP/1.1" 502 150 2025-12-11 19:58:16,085 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1b6915f407b7629e01c3'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:35580 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:58:16,841 - lava_callback - INFO - Received callback for node 693b1b6715f407b7629e01ba INFO: 172.21.0.1:35586 - "POST /node/693b1b6715f407b7629e01ba HTTP/1.0" 202 Accepted 2025-12-11 19:58:16,848 - lava_callback - INFO - Starting async job submit for node 693b1b6715f407b7629e01ba 2025-12-11 19:58:16,870 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:58:16,890 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1b6715f407b7629e01ba HTTP/1.1" 502 150 2025-12-11 19:58:16,890 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1b6715f407b7629e01ba'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:58:16,890 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1b6715f407b7629e01ba 2025-12-11 19:58:16,891 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1b6715f407b7629e01ba HTTP/1.1" 502 150 2025-12-11 19:58:16,891 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1b6715f407b7629e01ba'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:58:18,891 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1b6715f407b7629e01ba 2025-12-11 19:58:18,893 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1b6715f407b7629e01ba HTTP/1.1" 502 150 2025-12-11 19:58:18,893 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1b6715f407b7629e01ba'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:58:20,085 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1b6915f407b7629e01c3 2025-12-11 19:58:20,086 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1b6915f407b7629e01c3 HTTP/1.1" 502 150 2025-12-11 19:58:20,086 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1b6915f407b7629e01c3'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:58:22,893 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1b6715f407b7629e01ba 2025-12-11 19:58:22,894 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1b6715f407b7629e01ba HTTP/1.1" 502 150 2025-12-11 19:58:22,894 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1b6715f407b7629e01ba'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:48410 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48418 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:58:28,087 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1b6915f407b7629e01c3 2025-12-11 19:58:28,088 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1b6915f407b7629e01c3 HTTP/1.1" 502 150 2025-12-11 19:58:28,088 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1b6915f407b7629e01c3'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:58:30,895 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1b6715f407b7629e01ba 2025-12-11 19:58:30,896 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1b6715f407b7629e01ba HTTP/1.1" 502 150 2025-12-11 19:58:30,896 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1b6715f407b7629e01ba'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:49638 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49648 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:58:44,088 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1b6915f407b7629e01c3 2025-12-11 19:58:44,089 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1b6915f407b7629e01c3 HTTP/1.1" 502 150 2025-12-11 19:58:44,090 - lava_callback - ERROR - Error processing callback for node 693b1b6915f407b7629e01c3: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1b6915f407b7629e01c3 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1b6915f407b7629e01c3 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1b6915f407b7629e01c3 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:54560 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:58:46,896 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1b6715f407b7629e01ba 2025-12-11 19:58:46,897 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1b6715f407b7629e01ba HTTP/1.1" 502 150 2025-12-11 19:58:46,897 - lava_callback - ERROR - Error processing callback for node 693b1b6715f407b7629e01ba: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1b6715f407b7629e01ba (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1b6715f407b7629e01ba (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1b6715f407b7629e01ba (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:58:47,303 - lava_callback - INFO - Received callback for node 693afe05fee8383e92b6df76 INFO: 172.21.0.1:54572 - "POST /node/693afe05fee8383e92b6df76 HTTP/1.0" 202 Accepted 2025-12-11 19:58:47,310 - lava_callback - INFO - Starting async job submit for node 693afe05fee8383e92b6df76 2025-12-11 19:58:47,364 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:58:47,381 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afe05fee8383e92b6df76 HTTP/1.1" 502 150 2025-12-11 19:58:47,381 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afe05fee8383e92b6df76'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:58:47,381 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afe05fee8383e92b6df76 2025-12-11 19:58:47,382 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afe05fee8383e92b6df76 HTTP/1.1" 502 150 2025-12-11 19:58:47,382 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afe05fee8383e92b6df76'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:54578 - "GET /favicon.ico HTTP/1.0" 404 Not Found 2025-12-11 19:58:49,382 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afe05fee8383e92b6df76 2025-12-11 19:58:49,384 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afe05fee8383e92b6df76 HTTP/1.1" 502 150 2025-12-11 19:58:49,384 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afe05fee8383e92b6df76'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:58:53,384 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afe05fee8383e92b6df76 2025-12-11 19:58:53,386 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afe05fee8383e92b6df76 HTTP/1.1" 502 150 2025-12-11 19:58:53,386 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afe05fee8383e92b6df76'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:45710 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:59:00,834 - lava_callback - INFO - Received callback for node 6938874163d980a9748048a3 INFO: 172.21.0.1:40260 - "POST /node/6938874163d980a9748048a3 HTTP/1.0" 202 Accepted 2025-12-11 19:59:00,840 - lava_callback - INFO - Starting async job submit for node 6938874163d980a9748048a3 2025-12-11 19:59:00,873 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:59:00,883 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938874163d980a9748048a3 HTTP/1.1" 502 150 2025-12-11 19:59:00,883 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938874163d980a9748048a3'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:00,884 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938874163d980a9748048a3 2025-12-11 19:59:00,884 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938874163d980a9748048a3 HTTP/1.1" 502 150 2025-12-11 19:59:00,885 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938874163d980a9748048a3'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:01,386 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afe05fee8383e92b6df76 2025-12-11 19:59:01,387 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afe05fee8383e92b6df76 HTTP/1.1" 502 150 2025-12-11 19:59:01,387 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afe05fee8383e92b6df76'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:01,843 - lava_callback - INFO - Received callback for node 693b079bfee8383e92b70f06 INFO: 172.21.0.1:40264 - "POST /node/693b079bfee8383e92b70f06 HTTP/1.0" 202 Accepted 2025-12-11 19:59:01,852 - lava_callback - INFO - Starting async job submit for node 693b079bfee8383e92b70f06 2025-12-11 19:59:01,900 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:59:01,910 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b079bfee8383e92b70f06 HTTP/1.1" 502 150 2025-12-11 19:59:01,910 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b079bfee8383e92b70f06'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:01,910 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b079bfee8383e92b70f06 2025-12-11 19:59:01,911 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b079bfee8383e92b70f06 HTTP/1.1" 502 150 2025-12-11 19:59:01,911 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b079bfee8383e92b70f06'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:02,885 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938874163d980a9748048a3 2025-12-11 19:59:02,886 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938874163d980a9748048a3 HTTP/1.1" 502 150 2025-12-11 19:59:02,886 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938874163d980a9748048a3'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:03,911 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b079bfee8383e92b70f06 2025-12-11 19:59:03,912 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b079bfee8383e92b70f06 HTTP/1.1" 502 150 2025-12-11 19:59:03,913 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b079bfee8383e92b70f06'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:04,496 - lava_callback - INFO - Received callback for node 693ae6fafee8383e92b64f7e INFO: 172.21.0.1:40280 - "POST /node/693ae6fafee8383e92b64f7e HTTP/1.0" 202 Accepted 2025-12-11 19:59:04,503 - lava_callback - INFO - Starting async job submit for node 693ae6fafee8383e92b64f7e 2025-12-11 19:59:04,524 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:59:04,569 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae6fafee8383e92b64f7e HTTP/1.1" 502 150 2025-12-11 19:59:04,569 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae6fafee8383e92b64f7e'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:04,569 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae6fafee8383e92b64f7e 2025-12-11 19:59:04,569 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae6fafee8383e92b64f7e HTTP/1.1" 502 150 2025-12-11 19:59:04,569 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae6fafee8383e92b64f7e'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:05,351 - lava_callback - INFO - Received callback for node 693af46ffee8383e92b6ae57 INFO: 172.21.0.1:40290 - "POST /node/693af46ffee8383e92b6ae57 HTTP/1.0" 202 Accepted 2025-12-11 19:59:05,357 - lava_callback - INFO - Starting async job submit for node 693af46ffee8383e92b6ae57 2025-12-11 19:59:05,389 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:59:05,398 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af46ffee8383e92b6ae57 HTTP/1.1" 502 150 2025-12-11 19:59:05,398 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af46ffee8383e92b6ae57'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:05,398 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af46ffee8383e92b6ae57 2025-12-11 19:59:05,399 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af46ffee8383e92b6ae57 HTTP/1.1" 502 150 2025-12-11 19:59:05,399 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af46ffee8383e92b6ae57'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:40300 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:59:06,570 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae6fafee8383e92b64f7e 2025-12-11 19:59:06,571 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae6fafee8383e92b64f7e HTTP/1.1" 502 150 2025-12-11 19:59:06,571 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae6fafee8383e92b64f7e'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:06,887 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938874163d980a9748048a3 2025-12-11 19:59:06,888 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938874163d980a9748048a3 HTTP/1.1" 502 150 2025-12-11 19:59:06,888 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938874163d980a9748048a3'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:07,399 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af46ffee8383e92b6ae57 2025-12-11 19:59:07,400 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af46ffee8383e92b6ae57 HTTP/1.1" 502 150 2025-12-11 19:59:07,401 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af46ffee8383e92b6ae57'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:07,913 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b079bfee8383e92b70f06 2025-12-11 19:59:07,914 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b079bfee8383e92b70f06 HTTP/1.1" 502 150 2025-12-11 19:59:07,914 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b079bfee8383e92b70f06'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:10,571 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae6fafee8383e92b64f7e 2025-12-11 19:59:10,573 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae6fafee8383e92b64f7e HTTP/1.1" 502 150 2025-12-11 19:59:10,573 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae6fafee8383e92b64f7e'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:11,091 - lava_callback - INFO - Received callback for node 693b05ecfee8383e92b704b7 INFO: 172.21.0.1:60754 - "POST /node/693b05ecfee8383e92b704b7 HTTP/1.0" 202 Accepted 2025-12-11 19:59:11,106 - lava_callback - INFO - Starting async job submit for node 693b05ecfee8383e92b704b7 2025-12-11 19:59:11,142 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:59:11,179 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b05ecfee8383e92b704b7 HTTP/1.1" 502 150 2025-12-11 19:59:11,179 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b05ecfee8383e92b704b7'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:11,180 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b05ecfee8383e92b704b7 2025-12-11 19:59:11,181 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b05ecfee8383e92b704b7 HTTP/1.1" 502 150 2025-12-11 19:59:11,181 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b05ecfee8383e92b704b7'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:11,401 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af46ffee8383e92b6ae57 2025-12-11 19:59:11,402 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af46ffee8383e92b6ae57 HTTP/1.1" 502 150 2025-12-11 19:59:11,402 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af46ffee8383e92b6ae57'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:13,182 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b05ecfee8383e92b704b7 2025-12-11 19:59:13,183 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b05ecfee8383e92b704b7 HTTP/1.1" 502 150 2025-12-11 19:59:13,183 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b05ecfee8383e92b704b7'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:14,888 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938874163d980a9748048a3 2025-12-11 19:59:14,889 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938874163d980a9748048a3 HTTP/1.1" 502 150 2025-12-11 19:59:14,889 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938874163d980a9748048a3'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:60770 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 19:59:15,914 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b079bfee8383e92b70f06 2025-12-11 19:59:15,916 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b079bfee8383e92b70f06 HTTP/1.1" 502 150 2025-12-11 19:59:15,916 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b079bfee8383e92b70f06'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:60784 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:59:17,183 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b05ecfee8383e92b704b7 2025-12-11 19:59:17,184 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b05ecfee8383e92b704b7 HTTP/1.1" 502 150 2025-12-11 19:59:17,184 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b05ecfee8383e92b704b7'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:17,387 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afe05fee8383e92b6df76 2025-12-11 19:59:17,389 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afe05fee8383e92b6df76 HTTP/1.1" 502 150 2025-12-11 19:59:17,389 - lava_callback - ERROR - Error processing callback for node 693afe05fee8383e92b6df76: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afe05fee8383e92b6df76 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afe05fee8383e92b6df76 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afe05fee8383e92b6df76 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:59:18,145 - lava_callback - INFO - Received callback for node 693b15f7fee8383e92b748d4 INFO: 172.21.0.1:60794 - "POST /node/693b15f7fee8383e92b748d4 HTTP/1.0" 202 Accepted 2025-12-11 19:59:18,151 - lava_callback - INFO - Starting async job submit for node 693b15f7fee8383e92b748d4 2025-12-11 19:59:18,189 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:59:18,200 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b15f7fee8383e92b748d4 HTTP/1.1" 502 150 2025-12-11 19:59:18,200 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b15f7fee8383e92b748d4'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:18,200 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b15f7fee8383e92b748d4 2025-12-11 19:59:18,201 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b15f7fee8383e92b748d4 HTTP/1.1" 502 150 2025-12-11 19:59:18,201 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b15f7fee8383e92b748d4'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:18,574 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae6fafee8383e92b64f7e 2025-12-11 19:59:18,575 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae6fafee8383e92b64f7e HTTP/1.1" 502 150 2025-12-11 19:59:18,575 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae6fafee8383e92b64f7e'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:19,402 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af46ffee8383e92b6ae57 2025-12-11 19:59:19,404 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af46ffee8383e92b6ae57 HTTP/1.1" 502 150 2025-12-11 19:59:19,404 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af46ffee8383e92b6ae57'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:20,201 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b15f7fee8383e92b748d4 2025-12-11 19:59:20,203 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b15f7fee8383e92b748d4 HTTP/1.1" 502 150 2025-12-11 19:59:20,203 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b15f7fee8383e92b748d4'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:24,203 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b15f7fee8383e92b748d4 2025-12-11 19:59:24,205 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b15f7fee8383e92b748d4 HTTP/1.1" 502 150 2025-12-11 19:59:24,205 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b15f7fee8383e92b748d4'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:25,185 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b05ecfee8383e92b704b7 2025-12-11 19:59:25,186 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b05ecfee8383e92b704b7 HTTP/1.1" 502 150 2025-12-11 19:59:25,186 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b05ecfee8383e92b704b7'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:25,584 - lava_callback - INFO - Received callback for node 693b15b4fee8383e92b7480d INFO: 172.21.0.1:40308 - "POST /node/693b15b4fee8383e92b7480d HTTP/1.0" 202 Accepted 2025-12-11 19:59:25,591 - lava_callback - INFO - Starting async job submit for node 693b15b4fee8383e92b7480d 2025-12-11 19:59:25,627 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:59:25,637 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b15b4fee8383e92b7480d HTTP/1.1" 502 150 2025-12-11 19:59:25,637 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b15b4fee8383e92b7480d'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:25,637 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b15b4fee8383e92b7480d 2025-12-11 19:59:25,638 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b15b4fee8383e92b7480d HTTP/1.1" 502 150 2025-12-11 19:59:25,638 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b15b4fee8383e92b7480d'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:26,400 - lava_callback - INFO - Received callback for node 693ae856fee8383e92b658ba 2025-12-11 19:59:26,406 - lava_callback - INFO - Starting async job submit for node 693ae856fee8383e92b658ba INFO: 172.21.0.1:40312 - "POST /node/693ae856fee8383e92b658ba HTTP/1.0" 202 Accepted 2025-12-11 19:59:26,447 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:59:26,460 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae856fee8383e92b658ba HTTP/1.1" 502 150 2025-12-11 19:59:26,460 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae856fee8383e92b658ba'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:26,460 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae856fee8383e92b658ba 2025-12-11 19:59:26,461 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae856fee8383e92b658ba HTTP/1.1" 502 150 2025-12-11 19:59:26,461 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae856fee8383e92b658ba'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:40324 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40328 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:59:27,641 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b15b4fee8383e92b7480d 2025-12-11 19:59:27,643 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b15b4fee8383e92b7480d HTTP/1.1" 502 150 2025-12-11 19:59:27,643 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b15b4fee8383e92b7480d'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:28,461 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae856fee8383e92b658ba 2025-12-11 19:59:28,462 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae856fee8383e92b658ba HTTP/1.1" 502 150 2025-12-11 19:59:28,462 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae856fee8383e92b658ba'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:30,890 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938874163d980a9748048a3 2025-12-11 19:59:30,892 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938874163d980a9748048a3 HTTP/1.1" 502 150 2025-12-11 19:59:30,892 - lava_callback - ERROR - Error processing callback for node 6938874163d980a9748048a3: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938874163d980a9748048a3 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938874163d980a9748048a3 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938874163d980a9748048a3 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:59:31,643 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b15b4fee8383e92b7480d 2025-12-11 19:59:31,644 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b15b4fee8383e92b7480d HTTP/1.1" 502 150 2025-12-11 19:59:31,644 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b15b4fee8383e92b7480d'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:31,916 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b079bfee8383e92b70f06 2025-12-11 19:59:31,917 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b079bfee8383e92b70f06 HTTP/1.1" 502 150 2025-12-11 19:59:31,917 - lava_callback - ERROR - Error processing callback for node 693b079bfee8383e92b70f06: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b079bfee8383e92b70f06 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b079bfee8383e92b70f06 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b079bfee8383e92b70f06 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:59:32,205 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b15f7fee8383e92b748d4 2025-12-11 19:59:32,206 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b15f7fee8383e92b748d4 HTTP/1.1" 502 150 2025-12-11 19:59:32,206 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b15f7fee8383e92b748d4'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:32,463 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae856fee8383e92b658ba 2025-12-11 19:59:32,464 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae856fee8383e92b658ba HTTP/1.1" 502 150 2025-12-11 19:59:32,464 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae856fee8383e92b658ba'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:34,575 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae6fafee8383e92b64f7e 2025-12-11 19:59:34,576 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae6fafee8383e92b64f7e HTTP/1.1" 502 150 2025-12-11 19:59:34,576 - lava_callback - ERROR - Error processing callback for node 693ae6fafee8383e92b64f7e: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae6fafee8383e92b64f7e (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae6fafee8383e92b64f7e (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae6fafee8383e92b64f7e (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:59:35,404 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af46ffee8383e92b6ae57 2025-12-11 19:59:35,406 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af46ffee8383e92b6ae57 HTTP/1.1" 502 150 2025-12-11 19:59:35,406 - lava_callback - ERROR - Error processing callback for node 693af46ffee8383e92b6ae57: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af46ffee8383e92b6ae57 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af46ffee8383e92b6ae57 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af46ffee8383e92b6ae57 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:38074 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:59:39,644 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b15b4fee8383e92b7480d 2025-12-11 19:59:39,645 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b15b4fee8383e92b7480d HTTP/1.1" 502 150 2025-12-11 19:59:39,645 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b15b4fee8383e92b7480d'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:40,464 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae856fee8383e92b658ba 2025-12-11 19:59:40,465 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae856fee8383e92b658ba HTTP/1.1" 502 150 2025-12-11 19:59:40,466 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae856fee8383e92b658ba'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:41,186 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b05ecfee8383e92b704b7 2025-12-11 19:59:41,188 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b05ecfee8383e92b704b7 HTTP/1.1" 502 150 2025-12-11 19:59:41,188 - lava_callback - ERROR - Error processing callback for node 693b05ecfee8383e92b704b7: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b05ecfee8383e92b704b7 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b05ecfee8383e92b704b7 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b05ecfee8383e92b704b7 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:59:45,557 - lava_callback - INFO - Received callback for node 693af4e5fee8383e92b6b0bd INFO: 172.21.0.1:33738 - "POST /node/693af4e5fee8383e92b6b0bd HTTP/1.0" 202 Accepted 2025-12-11 19:59:45,567 - lava_callback - INFO - Starting async job submit for node 693af4e5fee8383e92b6b0bd 2025-12-11 19:59:45,586 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 19:59:45,595 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af4e5fee8383e92b6b0bd HTTP/1.1" 502 150 2025-12-11 19:59:45,595 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af4e5fee8383e92b6b0bd'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:45,595 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af4e5fee8383e92b6b0bd 2025-12-11 19:59:45,596 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af4e5fee8383e92b6b0bd HTTP/1.1" 502 150 2025-12-11 19:59:45,596 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af4e5fee8383e92b6b0bd'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:33748 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:59:47,596 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af4e5fee8383e92b6b0bd 2025-12-11 19:59:47,598 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af4e5fee8383e92b6b0bd HTTP/1.1" 502 150 2025-12-11 19:59:47,598 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af4e5fee8383e92b6b0bd'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:48,207 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b15f7fee8383e92b748d4 2025-12-11 19:59:48,208 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b15f7fee8383e92b748d4 HTTP/1.1" 502 150 2025-12-11 19:59:48,208 - lava_callback - ERROR - Error processing callback for node 693b15f7fee8383e92b748d4: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b15f7fee8383e92b748d4 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b15f7fee8383e92b748d4 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b15f7fee8383e92b748d4 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:59:51,598 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af4e5fee8383e92b6b0bd 2025-12-11 19:59:51,599 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af4e5fee8383e92b6b0bd HTTP/1.1" 502 150 2025-12-11 19:59:51,599 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af4e5fee8383e92b6b0bd'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 19:59:55,646 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b15b4fee8383e92b7480d 2025-12-11 19:59:55,647 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b15b4fee8383e92b7480d HTTP/1.1" 502 150 2025-12-11 19:59:55,648 - lava_callback - ERROR - Error processing callback for node 693b15b4fee8383e92b7480d: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b15b4fee8383e92b7480d (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b15b4fee8383e92b7480d (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b15b4fee8383e92b7480d (Caused by ResponseError('too many 502 error responses')) 2025-12-11 19:59:56,466 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae856fee8383e92b658ba 2025-12-11 19:59:56,476 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae856fee8383e92b658ba HTTP/1.1" 502 150 2025-12-11 19:59:56,476 - lava_callback - ERROR - Error processing callback for node 693ae856fee8383e92b658ba: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae856fee8383e92b658ba (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae856fee8383e92b658ba (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae856fee8383e92b658ba (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:35678 - "GET / HTTP/1.0" 200 OK 2025-12-11 19:59:59,599 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af4e5fee8383e92b6b0bd 2025-12-11 19:59:59,601 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af4e5fee8383e92b6b0bd HTTP/1.1" 502 150 2025-12-11 19:59:59,601 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af4e5fee8383e92b6b0bd'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:44476 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39338 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 20:00:15,601 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af4e5fee8383e92b6b0bd 2025-12-11 20:00:15,602 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af4e5fee8383e92b6b0bd HTTP/1.1" 502 150 2025-12-11 20:00:15,602 - lava_callback - ERROR - Error processing callback for node 693af4e5fee8383e92b6b0bd: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af4e5fee8383e92b6b0bd (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af4e5fee8383e92b6b0bd (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af4e5fee8383e92b6b0bd (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:39352 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:00:17,450 - lava_callback - INFO - Received callback for node 693b1b7415f407b7629e01ec INFO: 172.21.0.1:39356 - "POST /node/693b1b7415f407b7629e01ec HTTP/1.0" 202 Accepted 2025-12-11 20:00:17,457 - lava_callback - INFO - Starting async job submit for node 693b1b7415f407b7629e01ec 2025-12-11 20:00:17,479 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:00:17,492 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1b7415f407b7629e01ec HTTP/1.1" 502 150 2025-12-11 20:00:17,492 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1b7415f407b7629e01ec'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:17,492 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1b7415f407b7629e01ec 2025-12-11 20:00:17,493 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1b7415f407b7629e01ec HTTP/1.1" 502 150 2025-12-11 20:00:17,493 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1b7415f407b7629e01ec'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:17,699 - lava_callback - INFO - Received callback for node 693b1bdc15f407b7629e02f8 INFO: 172.21.0.1:39366 - "POST /node/693b1bdc15f407b7629e02f8 HTTP/1.0" 202 Accepted 2025-12-11 20:00:17,706 - lava_callback - INFO - Starting async job submit for node 693b1bdc15f407b7629e02f8 2025-12-11 20:00:17,741 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:00:17,752 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bdc15f407b7629e02f8 HTTP/1.1" 502 150 2025-12-11 20:00:17,752 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bdc15f407b7629e02f8'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:17,752 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bdc15f407b7629e02f8 2025-12-11 20:00:17,753 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bdc15f407b7629e02f8 HTTP/1.1" 502 150 2025-12-11 20:00:17,753 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bdc15f407b7629e02f8'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:19,493 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1b7415f407b7629e01ec 2025-12-11 20:00:19,494 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1b7415f407b7629e01ec HTTP/1.1" 502 150 2025-12-11 20:00:19,494 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1b7415f407b7629e01ec'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:19,753 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bdc15f407b7629e02f8 2025-12-11 20:00:19,755 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bdc15f407b7629e02f8 HTTP/1.1" 502 150 2025-12-11 20:00:19,755 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bdc15f407b7629e02f8'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:23,494 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1b7415f407b7629e01ec 2025-12-11 20:00:23,496 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1b7415f407b7629e01ec HTTP/1.1" 502 150 2025-12-11 20:00:23,496 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1b7415f407b7629e01ec'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:23,755 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bdc15f407b7629e02f8 2025-12-11 20:00:23,756 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bdc15f407b7629e02f8 HTTP/1.1" 502 150 2025-12-11 20:00:23,756 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bdc15f407b7629e02f8'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:25,980 - lava_callback - INFO - Received callback for node 693b15c4fee8383e92b74841 INFO: 172.21.0.1:56236 - "POST /node/693b15c4fee8383e92b74841 HTTP/1.0" 202 Accepted 2025-12-11 20:00:25,987 - lava_callback - INFO - Starting async job submit for node 693b15c4fee8383e92b74841 2025-12-11 20:00:26,024 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:00:26,042 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b15c4fee8383e92b74841 HTTP/1.1" 502 150 2025-12-11 20:00:26,042 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b15c4fee8383e92b74841'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:26,042 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b15c4fee8383e92b74841 2025-12-11 20:00:26,043 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b15c4fee8383e92b74841 HTTP/1.1" 502 150 2025-12-11 20:00:26,043 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b15c4fee8383e92b74841'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:56240 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56248 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:00:28,043 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b15c4fee8383e92b74841 2025-12-11 20:00:28,045 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b15c4fee8383e92b74841 HTTP/1.1" 502 150 2025-12-11 20:00:28,045 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b15c4fee8383e92b74841'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:31,496 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1b7415f407b7629e01ec 2025-12-11 20:00:31,497 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1b7415f407b7629e01ec HTTP/1.1" 502 150 2025-12-11 20:00:31,497 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1b7415f407b7629e01ec'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:31,657 - lava_callback - INFO - Received callback for node 693afe41fee8383e92b6e031 INFO: 172.21.0.1:34436 - "POST /node/693afe41fee8383e92b6e031 HTTP/1.0" 202 Accepted 2025-12-11 20:00:31,664 - lava_callback - INFO - Starting async job submit for node 693afe41fee8383e92b6e031 2025-12-11 20:00:31,736 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:00:31,756 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bdc15f407b7629e02f8 2025-12-11 20:00:31,757 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bdc15f407b7629e02f8 HTTP/1.1" 502 150 2025-12-11 20:00:31,757 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bdc15f407b7629e02f8'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:31,770 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afe41fee8383e92b6e031 HTTP/1.1" 502 150 2025-12-11 20:00:31,770 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afe41fee8383e92b6e031'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:31,770 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afe41fee8383e92b6e031 2025-12-11 20:00:31,771 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afe41fee8383e92b6e031 HTTP/1.1" 502 150 2025-12-11 20:00:31,771 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afe41fee8383e92b6e031'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:32,045 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b15c4fee8383e92b74841 2025-12-11 20:00:32,046 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b15c4fee8383e92b74841 HTTP/1.1" 502 150 2025-12-11 20:00:32,046 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b15c4fee8383e92b74841'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:33,771 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afe41fee8383e92b6e031 2025-12-11 20:00:33,773 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afe41fee8383e92b6e031 HTTP/1.1" 502 150 2025-12-11 20:00:33,773 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afe41fee8383e92b6e031'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:34444 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:00:37,773 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afe41fee8383e92b6e031 2025-12-11 20:00:37,774 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afe41fee8383e92b6e031 HTTP/1.1" 502 150 2025-12-11 20:00:37,775 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afe41fee8383e92b6e031'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:40,046 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b15c4fee8383e92b74841 2025-12-11 20:00:40,048 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b15c4fee8383e92b74841 HTTP/1.1" 502 150 2025-12-11 20:00:40,048 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b15c4fee8383e92b74841'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:45,775 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afe41fee8383e92b6e031 2025-12-11 20:00:45,776 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afe41fee8383e92b6e031 HTTP/1.1" 502 150 2025-12-11 20:00:45,776 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afe41fee8383e92b6e031'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:43942 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:00:47,429 - lava_callback - INFO - Received callback for node 693ae85bfee8383e92b658cb INFO: 172.21.0.1:43958 - "POST /node/693ae85bfee8383e92b658cb HTTP/1.0" 202 Accepted 2025-12-11 20:00:47,440 - lava_callback - INFO - Starting async job submit for node 693ae85bfee8383e92b658cb 2025-12-11 20:00:47,473 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:00:47,484 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae85bfee8383e92b658cb HTTP/1.1" 502 150 2025-12-11 20:00:47,484 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae85bfee8383e92b658cb'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:47,484 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae85bfee8383e92b658cb 2025-12-11 20:00:47,485 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae85bfee8383e92b658cb HTTP/1.1" 502 150 2025-12-11 20:00:47,485 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae85bfee8383e92b658cb'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:47,497 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1b7415f407b7629e01ec 2025-12-11 20:00:47,498 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1b7415f407b7629e01ec HTTP/1.1" 502 150 2025-12-11 20:00:47,499 - lava_callback - ERROR - Error processing callback for node 693b1b7415f407b7629e01ec: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1b7415f407b7629e01ec (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1b7415f407b7629e01ec (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1b7415f407b7629e01ec (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:00:47,621 - lava_callback - INFO - Received callback for node 693ae7f2fee8383e92b655ad INFO: 172.21.0.1:43962 - "POST /node/693ae7f2fee8383e92b655ad HTTP/1.0" 202 Accepted 2025-12-11 20:00:47,627 - lava_callback - INFO - Starting async job submit for node 693ae7f2fee8383e92b655ad 2025-12-11 20:00:47,649 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:00:47,661 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae7f2fee8383e92b655ad HTTP/1.1" 502 150 2025-12-11 20:00:47,661 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae7f2fee8383e92b655ad'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:47,662 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae7f2fee8383e92b655ad 2025-12-11 20:00:47,662 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae7f2fee8383e92b655ad HTTP/1.1" 502 150 2025-12-11 20:00:47,662 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae7f2fee8383e92b655ad'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:47,757 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bdc15f407b7629e02f8 2025-12-11 20:00:47,759 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bdc15f407b7629e02f8 HTTP/1.1" 502 150 2025-12-11 20:00:47,759 - lava_callback - ERROR - Error processing callback for node 693b1bdc15f407b7629e02f8: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1bdc15f407b7629e02f8 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1bdc15f407b7629e02f8 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1bdc15f407b7629e02f8 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:00:48,987 - lava_callback - INFO - Received callback for node 693ae85cfee8383e92b658d3 INFO: 172.21.0.1:43974 - "POST /node/693ae85cfee8383e92b658d3 HTTP/1.0" 202 Accepted 2025-12-11 20:00:48,997 - lava_callback - INFO - Starting async job submit for node 693ae85cfee8383e92b658d3 2025-12-11 20:00:49,039 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:00:49,056 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae85cfee8383e92b658d3 HTTP/1.1" 502 150 2025-12-11 20:00:49,056 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae85cfee8383e92b658d3'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:49,057 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae85cfee8383e92b658d3 2025-12-11 20:00:49,057 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae85cfee8383e92b658d3 HTTP/1.1" 502 150 2025-12-11 20:00:49,057 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae85cfee8383e92b658d3'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:49,485 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae85bfee8383e92b658cb 2025-12-11 20:00:49,487 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae85bfee8383e92b658cb HTTP/1.1" 502 150 2025-12-11 20:00:49,487 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae85bfee8383e92b658cb'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:49,663 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae7f2fee8383e92b655ad 2025-12-11 20:00:49,670 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae7f2fee8383e92b655ad HTTP/1.1" 502 150 2025-12-11 20:00:49,671 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae7f2fee8383e92b655ad'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:51,058 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae85cfee8383e92b658d3 2025-12-11 20:00:51,059 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae85cfee8383e92b658d3 HTTP/1.1" 502 150 2025-12-11 20:00:51,059 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae85cfee8383e92b658d3'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:53,008 - lava_callback - INFO - Received callback for node 69388a5863d980a974804930 INFO: 172.21.0.1:51764 - "POST /node/69388a5863d980a974804930 HTTP/1.0" 202 Accepted 2025-12-11 20:00:53,016 - lava_callback - INFO - Starting async job submit for node 69388a5863d980a974804930 2025-12-11 20:00:53,060 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:00:53,072 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69388a5863d980a974804930 HTTP/1.1" 502 150 2025-12-11 20:00:53,072 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/69388a5863d980a974804930'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:53,072 - urllib3.connectionpool - DEBUG - Retry: /latest/node/69388a5863d980a974804930 2025-12-11 20:00:53,073 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69388a5863d980a974804930 HTTP/1.1" 502 150 2025-12-11 20:00:53,073 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/69388a5863d980a974804930'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:53,487 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae85bfee8383e92b658cb 2025-12-11 20:00:53,488 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae85bfee8383e92b658cb HTTP/1.1" 502 150 2025-12-11 20:00:53,488 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae85bfee8383e92b658cb'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:53,671 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae7f2fee8383e92b655ad 2025-12-11 20:00:53,672 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae7f2fee8383e92b655ad HTTP/1.1" 502 150 2025-12-11 20:00:53,672 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae7f2fee8383e92b655ad'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:55,059 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae85cfee8383e92b658d3 2025-12-11 20:00:55,060 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae85cfee8383e92b658d3 HTTP/1.1" 502 150 2025-12-11 20:00:55,060 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae85cfee8383e92b658d3'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:55,073 - urllib3.connectionpool - DEBUG - Retry: /latest/node/69388a5863d980a974804930 2025-12-11 20:00:55,074 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69388a5863d980a974804930 HTTP/1.1" 502 150 2025-12-11 20:00:55,074 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/69388a5863d980a974804930'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:00:56,048 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b15c4fee8383e92b74841 2025-12-11 20:00:56,049 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b15c4fee8383e92b74841 HTTP/1.1" 502 150 2025-12-11 20:00:56,050 - lava_callback - ERROR - Error processing callback for node 693b15c4fee8383e92b74841: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b15c4fee8383e92b74841 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b15c4fee8383e92b74841 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b15c4fee8383e92b74841 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:51770 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:00:59,075 - urllib3.connectionpool - DEBUG - Retry: /latest/node/69388a5863d980a974804930 2025-12-11 20:00:59,083 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69388a5863d980a974804930 HTTP/1.1" 502 150 2025-12-11 20:00:59,083 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/69388a5863d980a974804930'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:01:01,489 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae85bfee8383e92b658cb 2025-12-11 20:01:01,490 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae85bfee8383e92b658cb HTTP/1.1" 502 150 2025-12-11 20:01:01,490 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae85bfee8383e92b658cb'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:01:01,673 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae7f2fee8383e92b655ad 2025-12-11 20:01:01,674 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae7f2fee8383e92b655ad HTTP/1.1" 502 150 2025-12-11 20:01:01,674 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae7f2fee8383e92b655ad'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:01:01,776 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afe41fee8383e92b6e031 2025-12-11 20:01:01,778 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afe41fee8383e92b6e031 HTTP/1.1" 502 150 2025-12-11 20:01:01,778 - lava_callback - ERROR - Error processing callback for node 693afe41fee8383e92b6e031: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afe41fee8383e92b6e031 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afe41fee8383e92b6e031 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afe41fee8383e92b6e031 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:01:03,061 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae85cfee8383e92b658d3 2025-12-11 20:01:03,062 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae85cfee8383e92b658d3 HTTP/1.1" 502 150 2025-12-11 20:01:03,062 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae85cfee8383e92b658d3'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:01:04,226 - lava_callback - INFO - Received callback for node 693b1a8d15f407b7629dfcd8 INFO: 172.21.0.1:58916 - "POST /node/693b1a8d15f407b7629dfcd8 HTTP/1.0" 202 Accepted 2025-12-11 20:01:04,234 - lava_callback - INFO - Starting async job submit for node 693b1a8d15f407b7629dfcd8 2025-12-11 20:01:04,267 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:01:04,278 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a8d15f407b7629dfcd8 HTTP/1.1" 502 150 2025-12-11 20:01:04,278 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a8d15f407b7629dfcd8'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:01:04,278 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a8d15f407b7629dfcd8 2025-12-11 20:01:04,279 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a8d15f407b7629dfcd8 HTTP/1.1" 502 150 2025-12-11 20:01:04,279 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a8d15f407b7629dfcd8'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:01:06,279 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a8d15f407b7629dfcd8 2025-12-11 20:01:06,280 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a8d15f407b7629dfcd8 HTTP/1.1" 502 150 2025-12-11 20:01:06,280 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a8d15f407b7629dfcd8'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:58930 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:01:07,084 - urllib3.connectionpool - DEBUG - Retry: /latest/node/69388a5863d980a974804930 2025-12-11 20:01:07,085 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69388a5863d980a974804930 HTTP/1.1" 502 150 2025-12-11 20:01:07,085 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/69388a5863d980a974804930'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:01:09,205 - lava_callback - INFO - Received callback for node 693b19c015f407b7629df6f6 INFO: 172.21.0.1:58942 - "POST /node/693b19c015f407b7629df6f6 HTTP/1.0" 202 Accepted 2025-12-11 20:01:09,211 - lava_callback - INFO - Starting async job submit for node 693b19c015f407b7629df6f6 2025-12-11 20:01:09,232 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:01:09,242 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19c015f407b7629df6f6 HTTP/1.1" 502 150 2025-12-11 20:01:09,242 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19c015f407b7629df6f6'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:01:09,242 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19c015f407b7629df6f6 2025-12-11 20:01:09,243 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19c015f407b7629df6f6 HTTP/1.1" 502 150 2025-12-11 20:01:09,243 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19c015f407b7629df6f6'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:01:10,281 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a8d15f407b7629dfcd8 2025-12-11 20:01:10,282 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a8d15f407b7629dfcd8 HTTP/1.1" 502 150 2025-12-11 20:01:10,282 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a8d15f407b7629dfcd8'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:01:11,243 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19c015f407b7629df6f6 2025-12-11 20:01:11,244 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19c015f407b7629df6f6 HTTP/1.1" 502 150 2025-12-11 20:01:11,244 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19c015f407b7629df6f6'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:51882 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 20:01:15,245 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19c015f407b7629df6f6 2025-12-11 20:01:15,246 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19c015f407b7629df6f6 HTTP/1.1" 502 150 2025-12-11 20:01:15,246 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19c015f407b7629df6f6'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:51896 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:01:17,490 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae85bfee8383e92b658cb 2025-12-11 20:01:17,492 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae85bfee8383e92b658cb HTTP/1.1" 502 150 2025-12-11 20:01:17,492 - lava_callback - ERROR - Error processing callback for node 693ae85bfee8383e92b658cb: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae85bfee8383e92b658cb (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae85bfee8383e92b658cb (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae85bfee8383e92b658cb (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:01:17,674 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae7f2fee8383e92b655ad 2025-12-11 20:01:17,676 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae7f2fee8383e92b655ad HTTP/1.1" 502 150 2025-12-11 20:01:17,676 - lava_callback - ERROR - Error processing callback for node 693ae7f2fee8383e92b655ad: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae7f2fee8383e92b655ad (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae7f2fee8383e92b655ad (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae7f2fee8383e92b655ad (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:01:18,282 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a8d15f407b7629dfcd8 2025-12-11 20:01:18,283 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a8d15f407b7629dfcd8 HTTP/1.1" 502 150 2025-12-11 20:01:18,284 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1a8d15f407b7629dfcd8'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:01:19,062 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae85cfee8383e92b658d3 2025-12-11 20:01:19,063 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae85cfee8383e92b658d3 HTTP/1.1" 502 150 2025-12-11 20:01:19,064 - lava_callback - ERROR - Error processing callback for node 693ae85cfee8383e92b658d3: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae85cfee8383e92b658d3 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae85cfee8383e92b658d3 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae85cfee8383e92b658d3 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:01:23,085 - urllib3.connectionpool - DEBUG - Retry: /latest/node/69388a5863d980a974804930 2025-12-11 20:01:23,087 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69388a5863d980a974804930 HTTP/1.1" 502 150 2025-12-11 20:01:23,087 - lava_callback - ERROR - Error processing callback for node 69388a5863d980a974804930: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/69388a5863d980a974804930 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/69388a5863d980a974804930 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/69388a5863d980a974804930 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:01:23,246 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19c015f407b7629df6f6 2025-12-11 20:01:23,248 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19c015f407b7629df6f6 HTTP/1.1" 502 150 2025-12-11 20:01:23,248 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19c015f407b7629df6f6'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:01:25,542 - lava_callback - INFO - Received callback for node 693ae859fee8383e92b658c4 INFO: 172.21.0.1:50524 - "POST /node/693ae859fee8383e92b658c4 HTTP/1.0" 202 Accepted 2025-12-11 20:01:25,548 - lava_callback - INFO - Starting async job submit for node 693ae859fee8383e92b658c4 2025-12-11 20:01:25,588 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:01:25,620 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae859fee8383e92b658c4 HTTP/1.1" 502 150 2025-12-11 20:01:25,621 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae859fee8383e92b658c4'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:01:25,621 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae859fee8383e92b658c4 2025-12-11 20:01:25,621 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae859fee8383e92b658c4 HTTP/1.1" 502 150 2025-12-11 20:01:25,621 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae859fee8383e92b658c4'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:50540 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50548 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:01:27,622 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae859fee8383e92b658c4 2025-12-11 20:01:27,623 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae859fee8383e92b658c4 HTTP/1.1" 502 150 2025-12-11 20:01:27,623 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae859fee8383e92b658c4'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:01:31,623 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae859fee8383e92b658c4 2025-12-11 20:01:31,629 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae859fee8383e92b658c4 HTTP/1.1" 502 150 2025-12-11 20:01:31,630 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae859fee8383e92b658c4'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:01:34,284 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1a8d15f407b7629dfcd8 2025-12-11 20:01:34,285 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1a8d15f407b7629dfcd8 HTTP/1.1" 502 150 2025-12-11 20:01:34,285 - lava_callback - ERROR - Error processing callback for node 693b1a8d15f407b7629dfcd8: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1a8d15f407b7629dfcd8 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1a8d15f407b7629dfcd8 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1a8d15f407b7629dfcd8 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:34356 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:01:39,248 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19c015f407b7629df6f6 2025-12-11 20:01:39,249 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19c015f407b7629df6f6 HTTP/1.1" 502 150 2025-12-11 20:01:39,249 - lava_callback - ERROR - Error processing callback for node 693b19c015f407b7629df6f6: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b19c015f407b7629df6f6 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b19c015f407b7629df6f6 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b19c015f407b7629df6f6 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:01:39,630 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae859fee8383e92b658c4 2025-12-11 20:01:39,631 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae859fee8383e92b658c4 HTTP/1.1" 502 150 2025-12-11 20:01:39,631 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae859fee8383e92b658c4'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:01:45,826 - lava_callback - INFO - Received callback for node 693b0a72fee8383e92b71a6e INFO: 172.21.0.1:50506 - "POST /node/693b0a72fee8383e92b71a6e HTTP/1.0" 202 Accepted 2025-12-11 20:01:45,833 - lava_callback - INFO - Starting async job submit for node 693b0a72fee8383e92b71a6e 2025-12-11 20:01:45,884 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:01:45,893 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0a72fee8383e92b71a6e HTTP/1.1" 502 150 2025-12-11 20:01:45,893 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0a72fee8383e92b71a6e'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:01:45,894 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0a72fee8383e92b71a6e 2025-12-11 20:01:45,894 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0a72fee8383e92b71a6e HTTP/1.1" 502 150 2025-12-11 20:01:45,894 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0a72fee8383e92b71a6e'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:50518 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:01:47,894 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0a72fee8383e92b71a6e 2025-12-11 20:01:47,896 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0a72fee8383e92b71a6e HTTP/1.1" 502 150 2025-12-11 20:01:47,896 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0a72fee8383e92b71a6e'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:01:51,896 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0a72fee8383e92b71a6e 2025-12-11 20:01:51,897 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0a72fee8383e92b71a6e HTTP/1.1" 502 150 2025-12-11 20:01:51,897 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0a72fee8383e92b71a6e'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:01:55,631 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae859fee8383e92b658c4 2025-12-11 20:01:55,633 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae859fee8383e92b658c4 HTTP/1.1" 502 150 2025-12-11 20:01:55,633 - lava_callback - ERROR - Error processing callback for node 693ae859fee8383e92b658c4: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae859fee8383e92b658c4 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae859fee8383e92b658c4 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae859fee8383e92b658c4 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:32982 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:01:59,898 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0a72fee8383e92b71a6e 2025-12-11 20:01:59,899 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0a72fee8383e92b71a6e HTTP/1.1" 502 150 2025-12-11 20:01:59,899 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0a72fee8383e92b71a6e'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:56038 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36498 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 20:02:15,899 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0a72fee8383e92b71a6e 2025-12-11 20:02:15,900 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0a72fee8383e92b71a6e HTTP/1.1" 502 150 2025-12-11 20:02:15,900 - lava_callback - ERROR - Error processing callback for node 693b0a72fee8383e92b71a6e: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0a72fee8383e92b71a6e (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0a72fee8383e92b71a6e (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0a72fee8383e92b71a6e (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:02:16,339 - lava_callback - INFO - Received callback for node 693b077dfee8383e92b70e75 INFO: 172.21.0.1:36510 - "POST /node/693b077dfee8383e92b70e75 HTTP/1.0" 202 Accepted 2025-12-11 20:02:16,347 - lava_callback - INFO - Starting async job submit for node 693b077dfee8383e92b70e75 2025-12-11 20:02:16,368 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:02:16,378 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b077dfee8383e92b70e75 HTTP/1.1" 502 150 2025-12-11 20:02:16,379 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b077dfee8383e92b70e75'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:02:16,379 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b077dfee8383e92b70e75 2025-12-11 20:02:16,379 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b077dfee8383e92b70e75 HTTP/1.1" 502 150 2025-12-11 20:02:16,380 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b077dfee8383e92b70e75'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:36512 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:02:18,380 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b077dfee8383e92b70e75 2025-12-11 20:02:18,381 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b077dfee8383e92b70e75 HTTP/1.1" 502 150 2025-12-11 20:02:18,381 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b077dfee8383e92b70e75'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:02:22,381 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b077dfee8383e92b70e75 2025-12-11 20:02:22,382 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b077dfee8383e92b70e75 HTTP/1.1" 502 150 2025-12-11 20:02:22,382 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b077dfee8383e92b70e75'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:02:22,721 - lava_callback - INFO - Received callback for node 693ae7f9fee8383e92b655cb INFO: 172.21.0.1:38498 - "POST /node/693ae7f9fee8383e92b655cb HTTP/1.0" 202 Accepted 2025-12-11 20:02:22,727 - lava_callback - INFO - Starting async job submit for node 693ae7f9fee8383e92b655cb 2025-12-11 20:02:22,749 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:02:22,779 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae7f9fee8383e92b655cb HTTP/1.1" 502 150 2025-12-11 20:02:22,779 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae7f9fee8383e92b655cb'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:02:22,780 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae7f9fee8383e92b655cb 2025-12-11 20:02:22,780 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae7f9fee8383e92b655cb HTTP/1.1" 502 150 2025-12-11 20:02:22,780 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae7f9fee8383e92b655cb'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:02:24,781 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae7f9fee8383e92b655cb 2025-12-11 20:02:24,782 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae7f9fee8383e92b655cb HTTP/1.1" 502 150 2025-12-11 20:02:24,782 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae7f9fee8383e92b655cb'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:38508 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38518 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:02:28,782 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae7f9fee8383e92b655cb 2025-12-11 20:02:28,783 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae7f9fee8383e92b655cb HTTP/1.1" 502 150 2025-12-11 20:02:28,783 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae7f9fee8383e92b655cb'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:02:30,382 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b077dfee8383e92b70e75 2025-12-11 20:02:30,384 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b077dfee8383e92b70e75 HTTP/1.1" 502 150 2025-12-11 20:02:30,384 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b077dfee8383e92b70e75'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:53688 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:02:36,783 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae7f9fee8383e92b655cb 2025-12-11 20:02:36,784 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae7f9fee8383e92b655cb HTTP/1.1" 502 150 2025-12-11 20:02:36,785 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae7f9fee8383e92b655cb'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:02:46,384 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b077dfee8383e92b70e75 2025-12-11 20:02:46,386 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b077dfee8383e92b70e75 HTTP/1.1" 502 150 2025-12-11 20:02:46,386 - lava_callback - ERROR - Error processing callback for node 693b077dfee8383e92b70e75: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b077dfee8383e92b70e75 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b077dfee8383e92b70e75 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b077dfee8383e92b70e75 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:42878 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:02:52,785 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae7f9fee8383e92b655cb 2025-12-11 20:02:52,786 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae7f9fee8383e92b655cb HTTP/1.1" 502 150 2025-12-11 20:02:52,786 - lava_callback - ERROR - Error processing callback for node 693ae7f9fee8383e92b655cb: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae7f9fee8383e92b655cb (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae7f9fee8383e92b655cb (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae7f9fee8383e92b655cb (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:51132 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:03:05,414 - lava_callback - INFO - Received callback for node 693af502fee8383e92b6b13f INFO: 172.21.0.1:39052 - "POST /node/693af502fee8383e92b6b13f HTTP/1.0" 202 Accepted 2025-12-11 20:03:05,420 - lava_callback - INFO - Starting async job submit for node 693af502fee8383e92b6b13f 2025-12-11 20:03:05,439 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:03:05,470 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af502fee8383e92b6b13f HTTP/1.1" 502 150 2025-12-11 20:03:05,470 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af502fee8383e92b6b13f'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:03:05,470 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af502fee8383e92b6b13f 2025-12-11 20:03:05,471 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af502fee8383e92b6b13f HTTP/1.1" 502 150 2025-12-11 20:03:05,471 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af502fee8383e92b6b13f'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:39066 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:03:07,471 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af502fee8383e92b6b13f 2025-12-11 20:03:07,473 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af502fee8383e92b6b13f HTTP/1.1" 502 150 2025-12-11 20:03:07,473 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af502fee8383e92b6b13f'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:03:11,473 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af502fee8383e92b6b13f 2025-12-11 20:03:11,474 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af502fee8383e92b6b13f HTTP/1.1" 502 150 2025-12-11 20:03:11,474 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af502fee8383e92b6b13f'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:44458 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:44474 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:03:16,577 - lava_callback - INFO - Received callback for node 693b19c015f407b7629df6f9 INFO: 172.21.0.1:44482 - "POST /node/693b19c015f407b7629df6f9 HTTP/1.0" 202 Accepted 2025-12-11 20:03:16,583 - lava_callback - INFO - Starting async job submit for node 693b19c015f407b7629df6f9 2025-12-11 20:03:16,610 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:03:16,625 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19c015f407b7629df6f9 HTTP/1.1" 502 150 2025-12-11 20:03:16,625 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19c015f407b7629df6f9'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:03:16,625 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19c015f407b7629df6f9 2025-12-11 20:03:16,626 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19c015f407b7629df6f9 HTTP/1.1" 502 150 2025-12-11 20:03:16,626 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19c015f407b7629df6f9'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:03:17,980 - lava_callback - INFO - Received callback for node 693b1c1a15f407b7629e0355 INFO: 172.21.0.1:44486 - "POST /node/693b1c1a15f407b7629e0355 HTTP/1.0" 202 Accepted 2025-12-11 20:03:17,991 - lava_callback - INFO - Starting async job submit for node 693b1c1a15f407b7629e0355 2025-12-11 20:03:18,026 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:03:18,037 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c1a15f407b7629e0355 HTTP/1.1" 502 150 2025-12-11 20:03:18,037 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c1a15f407b7629e0355'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:03:18,037 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c1a15f407b7629e0355 2025-12-11 20:03:18,038 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c1a15f407b7629e0355 HTTP/1.1" 502 150 2025-12-11 20:03:18,038 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c1a15f407b7629e0355'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:03:18,626 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19c015f407b7629df6f9 2025-12-11 20:03:18,632 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19c015f407b7629df6f9 HTTP/1.1" 502 150 2025-12-11 20:03:18,632 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19c015f407b7629df6f9'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:03:19,474 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af502fee8383e92b6b13f 2025-12-11 20:03:19,476 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af502fee8383e92b6b13f HTTP/1.1" 502 150 2025-12-11 20:03:19,476 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af502fee8383e92b6b13f'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:03:20,038 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c1a15f407b7629e0355 2025-12-11 20:03:20,040 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c1a15f407b7629e0355 HTTP/1.1" 502 150 2025-12-11 20:03:20,040 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c1a15f407b7629e0355'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:03:22,632 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19c015f407b7629df6f9 2025-12-11 20:03:22,633 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19c015f407b7629df6f9 HTTP/1.1" 502 150 2025-12-11 20:03:22,633 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19c015f407b7629df6f9'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:03:24,040 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c1a15f407b7629e0355 2025-12-11 20:03:24,041 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c1a15f407b7629e0355 HTTP/1.1" 502 150 2025-12-11 20:03:24,041 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c1a15f407b7629e0355'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:53258 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53264 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:03:30,292 - lava_callback - INFO - Received callback for node 693aff82fee8383e92b6e4b0 INFO: 172.21.0.1:60788 - "POST /node/693aff82fee8383e92b6e4b0 HTTP/1.0" 202 Accepted 2025-12-11 20:03:30,300 - lava_callback - INFO - Starting async job submit for node 693aff82fee8383e92b6e4b0 2025-12-11 20:03:30,353 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:03:30,382 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aff82fee8383e92b6e4b0 HTTP/1.1" 502 150 2025-12-11 20:03:30,382 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aff82fee8383e92b6e4b0'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:03:30,382 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aff82fee8383e92b6e4b0 2025-12-11 20:03:30,383 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aff82fee8383e92b6e4b0 HTTP/1.1" 502 150 2025-12-11 20:03:30,383 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aff82fee8383e92b6e4b0'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:03:30,633 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19c015f407b7629df6f9 2025-12-11 20:03:30,635 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19c015f407b7629df6f9 HTTP/1.1" 502 150 2025-12-11 20:03:30,635 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19c015f407b7629df6f9'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:03:32,041 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c1a15f407b7629e0355 2025-12-11 20:03:32,043 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c1a15f407b7629e0355 HTTP/1.1" 502 150 2025-12-11 20:03:32,043 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c1a15f407b7629e0355'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:03:32,383 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aff82fee8383e92b6e4b0 2025-12-11 20:03:32,384 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aff82fee8383e92b6e4b0 HTTP/1.1" 502 150 2025-12-11 20:03:32,385 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aff82fee8383e92b6e4b0'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:03:35,297 - lava_callback - INFO - Received callback for node 693b1ab415f407b7629dfd7e INFO: 172.21.0.1:60794 - "POST /node/693b1ab415f407b7629dfd7e HTTP/1.0" 202 Accepted 2025-12-11 20:03:35,305 - lava_callback - INFO - Starting async job submit for node 693b1ab415f407b7629dfd7e 2025-12-11 20:03:35,336 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:03:35,348 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ab415f407b7629dfd7e HTTP/1.1" 502 150 2025-12-11 20:03:35,348 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ab415f407b7629dfd7e'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:03:35,348 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ab415f407b7629dfd7e 2025-12-11 20:03:35,349 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ab415f407b7629dfd7e HTTP/1.1" 502 150 2025-12-11 20:03:35,349 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ab415f407b7629dfd7e'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:03:35,476 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af502fee8383e92b6b13f 2025-12-11 20:03:35,477 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af502fee8383e92b6b13f HTTP/1.1" 502 150 2025-12-11 20:03:35,478 - lava_callback - ERROR - Error processing callback for node 693af502fee8383e92b6b13f: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af502fee8383e92b6b13f (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af502fee8383e92b6b13f (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af502fee8383e92b6b13f (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:03:36,385 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aff82fee8383e92b6e4b0 2025-12-11 20:03:36,386 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aff82fee8383e92b6e4b0 HTTP/1.1" 502 150 2025-12-11 20:03:36,386 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aff82fee8383e92b6e4b0'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:60800 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:03:37,349 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ab415f407b7629dfd7e 2025-12-11 20:03:37,351 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ab415f407b7629dfd7e HTTP/1.1" 502 150 2025-12-11 20:03:37,351 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ab415f407b7629dfd7e'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:03:41,351 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ab415f407b7629dfd7e 2025-12-11 20:03:41,352 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ab415f407b7629dfd7e HTTP/1.1" 502 150 2025-12-11 20:03:41,352 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ab415f407b7629dfd7e'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:03:44,386 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aff82fee8383e92b6e4b0 2025-12-11 20:03:44,388 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aff82fee8383e92b6e4b0 HTTP/1.1" 502 150 2025-12-11 20:03:44,388 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aff82fee8383e92b6e4b0'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:58274 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:03:46,635 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19c015f407b7629df6f9 2025-12-11 20:03:46,636 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19c015f407b7629df6f9 HTTP/1.1" 502 150 2025-12-11 20:03:46,636 - lava_callback - ERROR - Error processing callback for node 693b19c015f407b7629df6f9: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b19c015f407b7629df6f9 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b19c015f407b7629df6f9 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b19c015f407b7629df6f9 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:03:48,043 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c1a15f407b7629e0355 2025-12-11 20:03:48,044 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c1a15f407b7629e0355 HTTP/1.1" 502 150 2025-12-11 20:03:48,044 - lava_callback - ERROR - Error processing callback for node 693b1c1a15f407b7629e0355: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1c1a15f407b7629e0355 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1c1a15f407b7629e0355 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1c1a15f407b7629e0355 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:03:49,352 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ab415f407b7629dfd7e 2025-12-11 20:03:49,355 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ab415f407b7629dfd7e HTTP/1.1" 502 150 2025-12-11 20:03:49,355 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ab415f407b7629dfd7e'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:39934 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:04:00,388 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aff82fee8383e92b6e4b0 2025-12-11 20:04:00,389 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aff82fee8383e92b6e4b0 HTTP/1.1" 502 150 2025-12-11 20:04:00,389 - lava_callback - ERROR - Error processing callback for node 693aff82fee8383e92b6e4b0: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aff82fee8383e92b6e4b0 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aff82fee8383e92b6e4b0 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aff82fee8383e92b6e4b0 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:04:05,356 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ab415f407b7629dfd7e 2025-12-11 20:04:05,357 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ab415f407b7629dfd7e HTTP/1.1" 502 150 2025-12-11 20:04:05,357 - lava_callback - ERROR - Error processing callback for node 693b1ab415f407b7629dfd7e: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1ab415f407b7629dfd7e (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1ab415f407b7629dfd7e (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1ab415f407b7629dfd7e (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:04:05,649 - lava_callback - INFO - Received callback for node 693b129cfee8383e92b73daa INFO: 172.21.0.1:50766 - "POST /node/693b129cfee8383e92b73daa HTTP/1.0" 202 Accepted 2025-12-11 20:04:05,657 - lava_callback - INFO - Starting async job submit for node 693b129cfee8383e92b73daa 2025-12-11 20:04:05,692 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:04:05,731 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b129cfee8383e92b73daa HTTP/1.1" 502 150 2025-12-11 20:04:05,731 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b129cfee8383e92b73daa'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:04:05,731 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b129cfee8383e92b73daa 2025-12-11 20:04:05,732 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b129cfee8383e92b73daa HTTP/1.1" 502 150 2025-12-11 20:04:05,732 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b129cfee8383e92b73daa'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:04:05,805 - lava_callback - INFO - Received callback for node 693ae7fbfee8383e92b655d4 INFO: 172.21.0.1:50770 - "POST /node/693ae7fbfee8383e92b655d4 HTTP/1.0" 202 Accepted 2025-12-11 20:04:05,814 - lava_callback - INFO - Starting async job submit for node 693ae7fbfee8383e92b655d4 2025-12-11 20:04:05,838 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:04:05,855 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae7fbfee8383e92b655d4 HTTP/1.1" 502 150 2025-12-11 20:04:05,855 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae7fbfee8383e92b655d4'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:04:05,855 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae7fbfee8383e92b655d4 2025-12-11 20:04:05,855 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae7fbfee8383e92b655d4 HTTP/1.1" 502 150 2025-12-11 20:04:05,855 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae7fbfee8383e92b655d4'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:50780 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:04:07,097 - lava_callback - INFO - Received callback for node 693b084efee8383e92b71284 INFO: 172.21.0.1:50782 - "POST /node/693b084efee8383e92b71284 HTTP/1.0" 202 Accepted 2025-12-11 20:04:07,108 - lava_callback - INFO - Starting async job submit for node 693b084efee8383e92b71284 2025-12-11 20:04:07,164 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:04:07,184 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b084efee8383e92b71284 HTTP/1.1" 502 150 2025-12-11 20:04:07,184 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b084efee8383e92b71284'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:04:07,184 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b084efee8383e92b71284 2025-12-11 20:04:07,185 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b084efee8383e92b71284 HTTP/1.1" 502 150 2025-12-11 20:04:07,185 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b084efee8383e92b71284'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:04:07,732 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b129cfee8383e92b73daa 2025-12-11 20:04:07,734 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b129cfee8383e92b73daa HTTP/1.1" 502 150 2025-12-11 20:04:07,734 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b129cfee8383e92b73daa'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:04:07,856 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae7fbfee8383e92b655d4 2025-12-11 20:04:07,857 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae7fbfee8383e92b655d4 HTTP/1.1" 502 150 2025-12-11 20:04:07,857 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae7fbfee8383e92b655d4'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:04:09,185 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b084efee8383e92b71284 2025-12-11 20:04:09,186 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b084efee8383e92b71284 HTTP/1.1" 502 150 2025-12-11 20:04:09,187 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b084efee8383e92b71284'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:04:11,735 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b129cfee8383e92b73daa 2025-12-11 20:04:11,736 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b129cfee8383e92b73daa HTTP/1.1" 502 150 2025-12-11 20:04:11,736 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b129cfee8383e92b73daa'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:04:11,857 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae7fbfee8383e92b655d4 2025-12-11 20:04:11,859 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae7fbfee8383e92b655d4 HTTP/1.1" 502 150 2025-12-11 20:04:11,859 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae7fbfee8383e92b655d4'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:04:13,187 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b084efee8383e92b71284 2025-12-11 20:04:13,188 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b084efee8383e92b71284 HTTP/1.1" 502 150 2025-12-11 20:04:13,188 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b084efee8383e92b71284'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:04:13,828 - lava_callback - INFO - Received callback for node 693b1c1a15f407b7629e0357 INFO: 172.21.0.1:47182 - "POST /node/693b1c1a15f407b7629e0357 HTTP/1.0" 202 Accepted 2025-12-11 20:04:13,839 - lava_callback - INFO - Starting async job submit for node 693b1c1a15f407b7629e0357 2025-12-11 20:04:13,878 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:04:13,911 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c1a15f407b7629e0357 HTTP/1.1" 502 150 2025-12-11 20:04:13,911 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c1a15f407b7629e0357'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:04:13,911 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c1a15f407b7629e0357 2025-12-11 20:04:13,912 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c1a15f407b7629e0357 HTTP/1.1" 502 150 2025-12-11 20:04:13,913 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c1a15f407b7629e0357'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:47194 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 20:04:15,913 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c1a15f407b7629e0357 2025-12-11 20:04:15,914 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c1a15f407b7629e0357 HTTP/1.1" 502 150 2025-12-11 20:04:15,914 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c1a15f407b7629e0357'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:47208 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:04:19,736 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b129cfee8383e92b73daa 2025-12-11 20:04:19,737 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b129cfee8383e92b73daa HTTP/1.1" 502 150 2025-12-11 20:04:19,738 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b129cfee8383e92b73daa'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:04:19,859 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae7fbfee8383e92b655d4 2025-12-11 20:04:19,860 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae7fbfee8383e92b655d4 HTTP/1.1" 502 150 2025-12-11 20:04:19,860 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae7fbfee8383e92b655d4'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:04:19,915 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c1a15f407b7629e0357 2025-12-11 20:04:19,916 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c1a15f407b7629e0357 HTTP/1.1" 502 150 2025-12-11 20:04:19,916 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c1a15f407b7629e0357'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:04:21,188 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b084efee8383e92b71284 2025-12-11 20:04:21,189 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b084efee8383e92b71284 HTTP/1.1" 502 150 2025-12-11 20:04:21,190 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b084efee8383e92b71284'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:39634 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39644 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:04:27,916 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c1a15f407b7629e0357 2025-12-11 20:04:27,917 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c1a15f407b7629e0357 HTTP/1.1" 502 150 2025-12-11 20:04:27,917 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c1a15f407b7629e0357'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:04:35,738 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b129cfee8383e92b73daa 2025-12-11 20:04:35,739 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b129cfee8383e92b73daa HTTP/1.1" 502 150 2025-12-11 20:04:35,739 - lava_callback - ERROR - Error processing callback for node 693b129cfee8383e92b73daa: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b129cfee8383e92b73daa (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b129cfee8383e92b73daa (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b129cfee8383e92b73daa (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:04:35,861 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae7fbfee8383e92b655d4 2025-12-11 20:04:35,862 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae7fbfee8383e92b655d4 HTTP/1.1" 502 150 2025-12-11 20:04:35,862 - lava_callback - ERROR - Error processing callback for node 693ae7fbfee8383e92b655d4: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae7fbfee8383e92b655d4 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae7fbfee8383e92b655d4 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae7fbfee8383e92b655d4 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:33270 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:04:37,190 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b084efee8383e92b71284 2025-12-11 20:04:37,191 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b084efee8383e92b71284 HTTP/1.1" 502 150 2025-12-11 20:04:37,191 - lava_callback - ERROR - Error processing callback for node 693b084efee8383e92b71284: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b084efee8383e92b71284 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b084efee8383e92b71284 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b084efee8383e92b71284 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:04:43,917 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c1a15f407b7629e0357 2025-12-11 20:04:43,919 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c1a15f407b7629e0357 HTTP/1.1" 502 150 2025-12-11 20:04:43,919 - lava_callback - ERROR - Error processing callback for node 693b1c1a15f407b7629e0357: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1c1a15f407b7629e0357 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1c1a15f407b7629e0357 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1c1a15f407b7629e0357 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:04:45,937 - lava_callback - INFO - Received callback for node 693b165d15f407b7629df1eb INFO: 172.21.0.1:57308 - "POST /node/693b165d15f407b7629df1eb HTTP/1.0" 202 Accepted 2025-12-11 20:04:45,948 - lava_callback - INFO - Starting async job submit for node 693b165d15f407b7629df1eb 2025-12-11 20:04:46,013 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:04:46,027 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b165d15f407b7629df1eb HTTP/1.1" 502 150 2025-12-11 20:04:46,027 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b165d15f407b7629df1eb'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:04:46,027 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b165d15f407b7629df1eb 2025-12-11 20:04:46,028 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b165d15f407b7629df1eb HTTP/1.1" 502 150 2025-12-11 20:04:46,029 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b165d15f407b7629df1eb'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:57318 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:04:48,029 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b165d15f407b7629df1eb 2025-12-11 20:04:48,030 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b165d15f407b7629df1eb HTTP/1.1" 502 150 2025-12-11 20:04:48,030 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b165d15f407b7629df1eb'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:04:52,030 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b165d15f407b7629df1eb 2025-12-11 20:04:52,032 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b165d15f407b7629df1eb HTTP/1.1" 502 150 2025-12-11 20:04:52,032 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b165d15f407b7629df1eb'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:52958 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:05:00,032 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b165d15f407b7629df1eb 2025-12-11 20:05:00,033 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b165d15f407b7629df1eb HTTP/1.1" 502 150 2025-12-11 20:05:00,033 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b165d15f407b7629df1eb'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:05:05,201 - lava_callback - INFO - Received callback for node 693ae9dcfee8383e92b65ea5 INFO: 172.21.0.1:45900 - "POST /node/693ae9dcfee8383e92b65ea5 HTTP/1.0" 202 Accepted 2025-12-11 20:05:05,253 - lava_callback - INFO - Starting async job submit for node 693ae9dcfee8383e92b65ea5 2025-12-11 20:05:05,278 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:05:05,296 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9dcfee8383e92b65ea5 HTTP/1.1" 502 150 2025-12-11 20:05:05,296 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9dcfee8383e92b65ea5'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:05:05,296 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9dcfee8383e92b65ea5 2025-12-11 20:05:05,297 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9dcfee8383e92b65ea5 HTTP/1.1" 502 150 2025-12-11 20:05:05,297 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9dcfee8383e92b65ea5'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:05:05,427 - lava_callback - INFO - Received callback for node 693af535fee8383e92b6b226 INFO: 172.21.0.1:45912 - "POST /node/693af535fee8383e92b6b226 HTTP/1.0" 202 Accepted 2025-12-11 20:05:05,432 - lava_callback - INFO - Starting async job submit for node 693af535fee8383e92b6b226 2025-12-11 20:05:05,449 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:05:05,459 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af535fee8383e92b6b226 HTTP/1.1" 502 150 2025-12-11 20:05:05,459 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af535fee8383e92b6b226'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:05:05,459 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af535fee8383e92b6b226 2025-12-11 20:05:05,460 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af535fee8383e92b6b226 HTTP/1.1" 502 150 2025-12-11 20:05:05,460 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af535fee8383e92b6b226'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:45928 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:05:07,297 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9dcfee8383e92b65ea5 2025-12-11 20:05:07,298 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9dcfee8383e92b65ea5 HTTP/1.1" 502 150 2025-12-11 20:05:07,298 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9dcfee8383e92b65ea5'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:05:07,460 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af535fee8383e92b6b226 2025-12-11 20:05:07,461 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af535fee8383e92b6b226 HTTP/1.1" 502 150 2025-12-11 20:05:07,461 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af535fee8383e92b6b226'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:05:11,298 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9dcfee8383e92b65ea5 2025-12-11 20:05:11,299 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9dcfee8383e92b65ea5 HTTP/1.1" 502 150 2025-12-11 20:05:11,299 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9dcfee8383e92b65ea5'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:05:11,461 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af535fee8383e92b6b226 2025-12-11 20:05:11,462 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af535fee8383e92b6b226 HTTP/1.1" 502 150 2025-12-11 20:05:11,462 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af535fee8383e92b6b226'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:05:12,384 - lava_callback - INFO - Received callback for node 693b077efee8383e92b70e7f INFO: 172.21.0.1:58478 - "POST /node/693b077efee8383e92b70e7f HTTP/1.0" 202 Accepted 2025-12-11 20:05:12,393 - lava_callback - INFO - Starting async job submit for node 693b077efee8383e92b70e7f 2025-12-11 20:05:12,413 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:05:12,424 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b077efee8383e92b70e7f HTTP/1.1" 502 150 2025-12-11 20:05:12,424 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b077efee8383e92b70e7f'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:05:12,424 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b077efee8383e92b70e7f 2025-12-11 20:05:12,425 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b077efee8383e92b70e7f HTTP/1.1" 502 150 2025-12-11 20:05:12,425 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b077efee8383e92b70e7f'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:05:14,425 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b077efee8383e92b70e7f 2025-12-11 20:05:14,426 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b077efee8383e92b70e7f HTTP/1.1" 502 150 2025-12-11 20:05:14,426 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b077efee8383e92b70e7f'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:58494 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 20:05:16,034 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b165d15f407b7629df1eb 2025-12-11 20:05:16,035 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b165d15f407b7629df1eb HTTP/1.1" 502 150 2025-12-11 20:05:16,035 - lava_callback - ERROR - Error processing callback for node 693b165d15f407b7629df1eb: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b165d15f407b7629df1eb (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b165d15f407b7629df1eb (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b165d15f407b7629df1eb (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:58506 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:05:18,426 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b077efee8383e92b70e7f 2025-12-11 20:05:18,428 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b077efee8383e92b70e7f HTTP/1.1" 502 150 2025-12-11 20:05:18,428 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b077efee8383e92b70e7f'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:05:19,300 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9dcfee8383e92b65ea5 2025-12-11 20:05:19,301 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9dcfee8383e92b65ea5 HTTP/1.1" 502 150 2025-12-11 20:05:19,301 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9dcfee8383e92b65ea5'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:05:19,463 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af535fee8383e92b6b226 2025-12-11 20:05:19,464 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af535fee8383e92b6b226 HTTP/1.1" 502 150 2025-12-11 20:05:19,464 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af535fee8383e92b6b226'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:05:24,131 - lava_callback - INFO - Received callback for node 693b09a0fee8383e92b717e3 INFO: 172.21.0.1:41976 - "POST /node/693b09a0fee8383e92b717e3 HTTP/1.0" 202 Accepted 2025-12-11 20:05:24,144 - lava_callback - INFO - Starting async job submit for node 693b09a0fee8383e92b717e3 2025-12-11 20:05:24,193 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:05:24,234 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b09a0fee8383e92b717e3 HTTP/1.1" 502 150 2025-12-11 20:05:24,234 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b09a0fee8383e92b717e3'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:05:24,234 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b09a0fee8383e92b717e3 2025-12-11 20:05:24,235 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b09a0fee8383e92b717e3 HTTP/1.1" 502 150 2025-12-11 20:05:24,235 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b09a0fee8383e92b717e3'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:05:26,235 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b09a0fee8383e92b717e3 2025-12-11 20:05:26,237 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b09a0fee8383e92b717e3 HTTP/1.1" 502 150 2025-12-11 20:05:26,237 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b09a0fee8383e92b717e3'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:05:26,428 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b077efee8383e92b70e7f 2025-12-11 20:05:26,429 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b077efee8383e92b70e7f HTTP/1.1" 502 150 2025-12-11 20:05:26,429 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b077efee8383e92b70e7f'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:41978 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41982 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:05:30,237 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b09a0fee8383e92b717e3 2025-12-11 20:05:30,238 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b09a0fee8383e92b717e3 HTTP/1.1" 502 150 2025-12-11 20:05:30,238 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b09a0fee8383e92b717e3'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:05:35,301 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9dcfee8383e92b65ea5 2025-12-11 20:05:35,302 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9dcfee8383e92b65ea5 HTTP/1.1" 502 150 2025-12-11 20:05:35,303 - lava_callback - ERROR - Error processing callback for node 693ae9dcfee8383e92b65ea5: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9dcfee8383e92b65ea5 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9dcfee8383e92b65ea5 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9dcfee8383e92b65ea5 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:05:35,464 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af535fee8383e92b6b226 2025-12-11 20:05:35,465 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af535fee8383e92b6b226 HTTP/1.1" 502 150 2025-12-11 20:05:35,465 - lava_callback - ERROR - Error processing callback for node 693af535fee8383e92b6b226: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af535fee8383e92b6b226 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af535fee8383e92b6b226 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af535fee8383e92b6b226 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:45038 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:05:38,239 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b09a0fee8383e92b717e3 2025-12-11 20:05:38,240 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b09a0fee8383e92b717e3 HTTP/1.1" 502 150 2025-12-11 20:05:38,240 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b09a0fee8383e92b717e3'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:05:42,430 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b077efee8383e92b70e7f 2025-12-11 20:05:42,431 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b077efee8383e92b70e7f HTTP/1.1" 502 150 2025-12-11 20:05:42,431 - lava_callback - ERROR - Error processing callback for node 693b077efee8383e92b70e7f: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b077efee8383e92b70e7f (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b077efee8383e92b70e7f (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b077efee8383e92b70e7f (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:05:45,713 - lava_callback - INFO - Received callback for node 693ae863fee8383e92b658ef INFO: 172.21.0.1:33932 - "POST /node/693ae863fee8383e92b658ef HTTP/1.0" 202 Accepted 2025-12-11 20:05:45,719 - lava_callback - INFO - Starting async job submit for node 693ae863fee8383e92b658ef 2025-12-11 20:05:45,776 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:05:45,792 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae863fee8383e92b658ef HTTP/1.1" 502 150 2025-12-11 20:05:45,792 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae863fee8383e92b658ef'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:05:45,792 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae863fee8383e92b658ef 2025-12-11 20:05:45,793 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae863fee8383e92b658ef HTTP/1.1" 502 150 2025-12-11 20:05:45,793 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae863fee8383e92b658ef'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:33948 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:05:47,793 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae863fee8383e92b658ef 2025-12-11 20:05:47,794 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae863fee8383e92b658ef HTTP/1.1" 502 150 2025-12-11 20:05:47,795 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae863fee8383e92b658ef'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:05:48,891 - lava_callback - INFO - Received callback for node 693ae98bfee8383e92b65d62 INFO: 172.21.0.1:33950 - "POST /node/693ae98bfee8383e92b65d62 HTTP/1.0" 202 Accepted 2025-12-11 20:05:48,901 - lava_callback - INFO - Starting async job submit for node 693ae98bfee8383e92b65d62 2025-12-11 20:05:48,966 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:05:49,010 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae98bfee8383e92b65d62 HTTP/1.1" 502 150 2025-12-11 20:05:49,010 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae98bfee8383e92b65d62'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:05:49,010 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae98bfee8383e92b65d62 2025-12-11 20:05:49,011 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae98bfee8383e92b65d62 HTTP/1.1" 502 150 2025-12-11 20:05:49,011 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae98bfee8383e92b65d62'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:05:51,011 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae98bfee8383e92b65d62 2025-12-11 20:05:51,012 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae98bfee8383e92b65d62 HTTP/1.1" 502 150 2025-12-11 20:05:51,013 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae98bfee8383e92b65d62'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:05:51,795 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae863fee8383e92b658ef 2025-12-11 20:05:51,796 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae863fee8383e92b658ef HTTP/1.1" 502 150 2025-12-11 20:05:51,796 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae863fee8383e92b658ef'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:05:54,241 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b09a0fee8383e92b717e3 2025-12-11 20:05:54,242 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b09a0fee8383e92b717e3 HTTP/1.1" 502 150 2025-12-11 20:05:54,242 - lava_callback - ERROR - Error processing callback for node 693b09a0fee8383e92b717e3: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b09a0fee8383e92b717e3 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b09a0fee8383e92b717e3 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b09a0fee8383e92b717e3 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:05:55,013 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae98bfee8383e92b65d62 2025-12-11 20:05:55,014 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae98bfee8383e92b65d62 HTTP/1.1" 502 150 2025-12-11 20:05:55,014 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae98bfee8383e92b65d62'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:36620 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:05:59,796 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae863fee8383e92b658ef 2025-12-11 20:05:59,797 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae863fee8383e92b658ef HTTP/1.1" 502 150 2025-12-11 20:05:59,798 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae863fee8383e92b658ef'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:06:03,014 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae98bfee8383e92b65d62 2025-12-11 20:06:03,016 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae98bfee8383e92b65d62 HTTP/1.1" 502 150 2025-12-11 20:06:03,016 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae98bfee8383e92b65d62'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:06:05,841 - lava_callback - INFO - Received callback for node 693b165d15f407b7629df1ed INFO: 172.21.0.1:42670 - "POST /node/693b165d15f407b7629df1ed HTTP/1.0" 202 Accepted 2025-12-11 20:06:05,848 - lava_callback - INFO - Starting async job submit for node 693b165d15f407b7629df1ed 2025-12-11 20:06:05,883 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:06:05,893 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b165d15f407b7629df1ed HTTP/1.1" 502 150 2025-12-11 20:06:05,893 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b165d15f407b7629df1ed'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:06:05,893 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b165d15f407b7629df1ed 2025-12-11 20:06:05,894 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b165d15f407b7629df1ed HTTP/1.1" 502 150 2025-12-11 20:06:05,894 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b165d15f407b7629df1ed'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:42684 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:06:07,894 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b165d15f407b7629df1ed 2025-12-11 20:06:07,895 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b165d15f407b7629df1ed HTTP/1.1" 502 150 2025-12-11 20:06:07,895 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b165d15f407b7629df1ed'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:06:11,896 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b165d15f407b7629df1ed 2025-12-11 20:06:11,897 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b165d15f407b7629df1ed HTTP/1.1" 502 150 2025-12-11 20:06:11,897 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b165d15f407b7629df1ed'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:39170 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 20:06:15,798 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae863fee8383e92b658ef 2025-12-11 20:06:15,799 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae863fee8383e92b658ef HTTP/1.1" 502 150 2025-12-11 20:06:15,799 - lava_callback - ERROR - Error processing callback for node 693ae863fee8383e92b658ef: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae863fee8383e92b658ef (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae863fee8383e92b658ef (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae863fee8383e92b658ef (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:39184 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:06:19,016 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae98bfee8383e92b65d62 2025-12-11 20:06:19,017 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae98bfee8383e92b65d62 HTTP/1.1" 502 150 2025-12-11 20:06:19,018 - lava_callback - ERROR - Error processing callback for node 693ae98bfee8383e92b65d62: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae98bfee8383e92b65d62 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae98bfee8383e92b65d62 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae98bfee8383e92b65d62 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:06:19,897 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b165d15f407b7629df1ed 2025-12-11 20:06:19,899 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b165d15f407b7629df1ed HTTP/1.1" 502 150 2025-12-11 20:06:19,899 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b165d15f407b7629df1ed'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:06:21,198 - lava_callback - INFO - Received callback for node 693b19c115f407b7629df6fc INFO: 172.21.0.1:50592 - "POST /node/693b19c115f407b7629df6fc HTTP/1.0" 202 Accepted 2025-12-11 20:06:21,205 - lava_callback - INFO - Starting async job submit for node 693b19c115f407b7629df6fc 2025-12-11 20:06:21,228 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:06:21,249 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19c115f407b7629df6fc HTTP/1.1" 502 150 2025-12-11 20:06:21,249 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19c115f407b7629df6fc'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:06:21,249 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19c115f407b7629df6fc 2025-12-11 20:06:21,249 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19c115f407b7629df6fc HTTP/1.1" 502 150 2025-12-11 20:06:21,250 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19c115f407b7629df6fc'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:06:23,250 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19c115f407b7629df6fc 2025-12-11 20:06:23,251 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19c115f407b7629df6fc HTTP/1.1" 502 150 2025-12-11 20:06:23,251 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19c115f407b7629df6fc'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:50608 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50620 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:06:27,251 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19c115f407b7629df6fc 2025-12-11 20:06:27,252 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19c115f407b7629df6fc HTTP/1.1" 502 150 2025-12-11 20:06:27,252 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19c115f407b7629df6fc'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:06:35,252 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19c115f407b7629df6fc 2025-12-11 20:06:35,254 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19c115f407b7629df6fc HTTP/1.1" 502 150 2025-12-11 20:06:35,254 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19c115f407b7629df6fc'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:06:35,899 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b165d15f407b7629df1ed 2025-12-11 20:06:35,900 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b165d15f407b7629df1ed HTTP/1.1" 502 150 2025-12-11 20:06:35,901 - lava_callback - ERROR - Error processing callback for node 693b165d15f407b7629df1ed: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b165d15f407b7629df1ed (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b165d15f407b7629df1ed (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b165d15f407b7629df1ed (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:36656 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:06:46,103 - lava_callback - INFO - Received callback for node 693b0b3efee8383e92b71c80 INFO: 172.21.0.1:50568 - "POST /node/693b0b3efee8383e92b71c80 HTTP/1.0" 202 Accepted 2025-12-11 20:06:46,109 - lava_callback - INFO - Starting async job submit for node 693b0b3efee8383e92b71c80 2025-12-11 20:06:46,163 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:06:46,173 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0b3efee8383e92b71c80 HTTP/1.1" 502 150 2025-12-11 20:06:46,173 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0b3efee8383e92b71c80'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:06:46,174 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0b3efee8383e92b71c80 2025-12-11 20:06:46,174 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0b3efee8383e92b71c80 HTTP/1.1" 502 150 2025-12-11 20:06:46,174 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0b3efee8383e92b71c80'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:50572 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:06:46,665 - lava_callback - INFO - Received callback for node 693ae85efee8383e92b658d8 INFO: 172.21.0.1:50578 - "POST /node/693ae85efee8383e92b658d8 HTTP/1.0" 202 Accepted 2025-12-11 20:06:46,672 - lava_callback - INFO - Starting async job submit for node 693ae85efee8383e92b658d8 2025-12-11 20:06:46,705 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:06:46,715 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae85efee8383e92b658d8 HTTP/1.1" 502 150 2025-12-11 20:06:46,715 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae85efee8383e92b658d8'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:06:46,716 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae85efee8383e92b658d8 2025-12-11 20:06:46,716 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae85efee8383e92b658d8 HTTP/1.1" 502 150 2025-12-11 20:06:46,716 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae85efee8383e92b658d8'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:06:47,658 - lava_callback - INFO - Received callback for node 693b165d15f407b7629df1ef INFO: 172.21.0.1:50580 - "POST /node/693b165d15f407b7629df1ef HTTP/1.0" 202 Accepted 2025-12-11 20:06:47,664 - lava_callback - INFO - Starting async job submit for node 693b165d15f407b7629df1ef 2025-12-11 20:06:47,687 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:06:47,696 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b165d15f407b7629df1ef HTTP/1.1" 502 150 2025-12-11 20:06:47,696 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b165d15f407b7629df1ef'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:06:47,696 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b165d15f407b7629df1ef 2025-12-11 20:06:47,697 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b165d15f407b7629df1ef HTTP/1.1" 502 150 2025-12-11 20:06:47,697 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b165d15f407b7629df1ef'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:06:48,175 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0b3efee8383e92b71c80 2025-12-11 20:06:48,176 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0b3efee8383e92b71c80 HTTP/1.1" 502 150 2025-12-11 20:06:48,176 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0b3efee8383e92b71c80'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:06:48,717 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae85efee8383e92b658d8 2025-12-11 20:06:48,718 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae85efee8383e92b658d8 HTTP/1.1" 502 150 2025-12-11 20:06:48,718 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae85efee8383e92b658d8'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:06:49,697 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b165d15f407b7629df1ef 2025-12-11 20:06:49,703 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b165d15f407b7629df1ef HTTP/1.1" 502 150 2025-12-11 20:06:49,703 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b165d15f407b7629df1ef'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:06:51,254 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19c115f407b7629df6fc 2025-12-11 20:06:51,256 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19c115f407b7629df6fc HTTP/1.1" 502 150 2025-12-11 20:06:51,256 - lava_callback - ERROR - Error processing callback for node 693b19c115f407b7629df6fc: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b19c115f407b7629df6fc (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b19c115f407b7629df6fc (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b19c115f407b7629df6fc (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:06:52,176 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0b3efee8383e92b71c80 2025-12-11 20:06:52,178 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0b3efee8383e92b71c80 HTTP/1.1" 502 150 2025-12-11 20:06:52,178 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0b3efee8383e92b71c80'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:06:52,718 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae85efee8383e92b658d8 2025-12-11 20:06:52,719 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae85efee8383e92b658d8 HTTP/1.1" 502 150 2025-12-11 20:06:52,719 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae85efee8383e92b658d8'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:06:53,703 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b165d15f407b7629df1ef 2025-12-11 20:06:53,704 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b165d15f407b7629df1ef HTTP/1.1" 502 150 2025-12-11 20:06:53,704 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b165d15f407b7629df1ef'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:37696 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:07:00,178 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0b3efee8383e92b71c80 2025-12-11 20:07:00,179 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0b3efee8383e92b71c80 HTTP/1.1" 502 150 2025-12-11 20:07:00,179 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0b3efee8383e92b71c80'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:07:00,720 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae85efee8383e92b658d8 2025-12-11 20:07:00,721 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae85efee8383e92b658d8 HTTP/1.1" 502 150 2025-12-11 20:07:00,721 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae85efee8383e92b658d8'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:07:01,704 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b165d15f407b7629df1ef 2025-12-11 20:07:01,706 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b165d15f407b7629df1ef HTTP/1.1" 502 150 2025-12-11 20:07:01,706 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b165d15f407b7629df1ef'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:51698 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51032 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 20:07:16,180 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0b3efee8383e92b71c80 2025-12-11 20:07:16,181 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0b3efee8383e92b71c80 HTTP/1.1" 502 150 2025-12-11 20:07:16,181 - lava_callback - ERROR - Error processing callback for node 693b0b3efee8383e92b71c80: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0b3efee8383e92b71c80 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0b3efee8383e92b71c80 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0b3efee8383e92b71c80 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:51040 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:07:16,721 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae85efee8383e92b658d8 2025-12-11 20:07:16,722 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae85efee8383e92b658d8 HTTP/1.1" 502 150 2025-12-11 20:07:16,722 - lava_callback - ERROR - Error processing callback for node 693ae85efee8383e92b658d8: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae85efee8383e92b658d8 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae85efee8383e92b658d8 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae85efee8383e92b658d8 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:07:17,706 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b165d15f407b7629df1ef 2025-12-11 20:07:17,707 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b165d15f407b7629df1ef HTTP/1.1" 502 150 2025-12-11 20:07:17,708 - lava_callback - ERROR - Error processing callback for node 693b165d15f407b7629df1ef: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b165d15f407b7629df1ef (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b165d15f407b7629df1ef (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b165d15f407b7629df1ef (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:07:23,949 - lava_callback - INFO - Received callback for node 693ae9aafee8383e92b65dda INFO: 172.21.0.1:60784 - "POST /node/693ae9aafee8383e92b65dda HTTP/1.0" 202 Accepted 2025-12-11 20:07:23,959 - lava_callback - INFO - Starting async job submit for node 693ae9aafee8383e92b65dda 2025-12-11 20:07:23,980 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:07:23,990 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9aafee8383e92b65dda HTTP/1.1" 502 150 2025-12-11 20:07:23,990 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9aafee8383e92b65dda'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:07:23,990 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9aafee8383e92b65dda 2025-12-11 20:07:23,991 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9aafee8383e92b65dda HTTP/1.1" 502 150 2025-12-11 20:07:23,991 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9aafee8383e92b65dda'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:07:25,991 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9aafee8383e92b65dda 2025-12-11 20:07:25,992 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9aafee8383e92b65dda HTTP/1.1" 502 150 2025-12-11 20:07:25,992 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9aafee8383e92b65dda'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:60800 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60816 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:07:29,993 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9aafee8383e92b65dda 2025-12-11 20:07:29,994 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9aafee8383e92b65dda HTTP/1.1" 502 150 2025-12-11 20:07:29,994 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9aafee8383e92b65dda'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:59034 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:07:37,994 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9aafee8383e92b65dda 2025-12-11 20:07:37,996 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9aafee8383e92b65dda HTTP/1.1" 502 150 2025-12-11 20:07:37,996 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9aafee8383e92b65dda'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:44588 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:07:53,996 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9aafee8383e92b65dda 2025-12-11 20:07:53,997 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9aafee8383e92b65dda HTTP/1.1" 502 150 2025-12-11 20:07:53,997 - lava_callback - ERROR - Error processing callback for node 693ae9aafee8383e92b65dda: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9aafee8383e92b65dda (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9aafee8383e92b65dda (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9aafee8383e92b65dda (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:59594 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:08:05,738 - lava_callback - INFO - Received callback for node 693af590fee8383e92b6b42e INFO: 172.21.0.1:44612 - "POST /node/693af590fee8383e92b6b42e HTTP/1.0" 202 Accepted 2025-12-11 20:08:05,745 - lava_callback - INFO - Starting async job submit for node 693af590fee8383e92b6b42e 2025-12-11 20:08:05,765 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:08:05,784 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af590fee8383e92b6b42e HTTP/1.1" 502 150 2025-12-11 20:08:05,784 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af590fee8383e92b6b42e'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:08:05,784 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af590fee8383e92b6b42e 2025-12-11 20:08:05,785 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af590fee8383e92b6b42e HTTP/1.1" 502 150 2025-12-11 20:08:05,785 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af590fee8383e92b6b42e'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:44618 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:08:07,277 - lava_callback - INFO - Received callback for node 693b16fb15f407b7629df285 INFO: 172.21.0.1:44622 - "POST /node/693b16fb15f407b7629df285 HTTP/1.0" 202 Accepted 2025-12-11 20:08:07,286 - lava_callback - INFO - Starting async job submit for node 693b16fb15f407b7629df285 2025-12-11 20:08:07,322 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:08:07,359 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b16fb15f407b7629df285 HTTP/1.1" 502 150 2025-12-11 20:08:07,359 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b16fb15f407b7629df285'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:08:07,359 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b16fb15f407b7629df285 2025-12-11 20:08:07,360 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b16fb15f407b7629df285 HTTP/1.1" 502 150 2025-12-11 20:08:07,360 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b16fb15f407b7629df285'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:08:07,785 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af590fee8383e92b6b42e 2025-12-11 20:08:07,786 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af590fee8383e92b6b42e HTTP/1.1" 502 150 2025-12-11 20:08:07,786 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af590fee8383e92b6b42e'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:08:09,360 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b16fb15f407b7629df285 2025-12-11 20:08:09,361 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b16fb15f407b7629df285 HTTP/1.1" 502 150 2025-12-11 20:08:09,362 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b16fb15f407b7629df285'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:08:11,787 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af590fee8383e92b6b42e 2025-12-11 20:08:11,788 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af590fee8383e92b6b42e HTTP/1.1" 502 150 2025-12-11 20:08:11,788 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af590fee8383e92b6b42e'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:08:12,357 - lava_callback - INFO - Received callback for node 693b077efee8383e92b70e81 INFO: 172.21.0.1:48852 - "POST /node/693b077efee8383e92b70e81 HTTP/1.0" 202 Accepted 2025-12-11 20:08:12,374 - lava_callback - INFO - Starting async job submit for node 693b077efee8383e92b70e81 2025-12-11 20:08:12,418 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:08:12,432 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b077efee8383e92b70e81 HTTP/1.1" 502 150 2025-12-11 20:08:12,433 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b077efee8383e92b70e81'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:08:12,433 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b077efee8383e92b70e81 2025-12-11 20:08:12,434 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b077efee8383e92b70e81 HTTP/1.1" 502 150 2025-12-11 20:08:12,434 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b077efee8383e92b70e81'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:08:13,362 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b16fb15f407b7629df285 2025-12-11 20:08:13,363 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b16fb15f407b7629df285 HTTP/1.1" 502 150 2025-12-11 20:08:13,363 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b16fb15f407b7629df285'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:08:14,434 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b077efee8383e92b70e81 2025-12-11 20:08:14,435 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b077efee8383e92b70e81 HTTP/1.1" 502 150 2025-12-11 20:08:14,436 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b077efee8383e92b70e81'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:48856 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48872 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:08:18,436 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b077efee8383e92b70e81 2025-12-11 20:08:18,437 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b077efee8383e92b70e81 HTTP/1.1" 502 150 2025-12-11 20:08:18,437 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b077efee8383e92b70e81'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:08:19,788 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af590fee8383e92b6b42e 2025-12-11 20:08:19,789 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af590fee8383e92b6b42e HTTP/1.1" 502 150 2025-12-11 20:08:19,789 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af590fee8383e92b6b42e'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:08:21,364 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b16fb15f407b7629df285 2025-12-11 20:08:21,365 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b16fb15f407b7629df285 HTTP/1.1" 502 150 2025-12-11 20:08:21,365 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b16fb15f407b7629df285'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:08:26,203 - lava_callback - INFO - Received callback for node 693b0b8cfee8383e92b71d10 INFO: 172.21.0.1:55270 - "POST /node/693b0b8cfee8383e92b71d10 HTTP/1.0" 202 Accepted 2025-12-11 20:08:26,214 - lava_callback - INFO - Starting async job submit for node 693b0b8cfee8383e92b71d10 2025-12-11 20:08:26,254 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:08:26,287 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0b8cfee8383e92b71d10 HTTP/1.1" 502 150 2025-12-11 20:08:26,287 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0b8cfee8383e92b71d10'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:08:26,288 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0b8cfee8383e92b71d10 2025-12-11 20:08:26,289 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0b8cfee8383e92b71d10 HTTP/1.1" 502 150 2025-12-11 20:08:26,289 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0b8cfee8383e92b71d10'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:08:26,437 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b077efee8383e92b70e81 2025-12-11 20:08:26,438 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b077efee8383e92b70e81 HTTP/1.1" 502 150 2025-12-11 20:08:26,438 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b077efee8383e92b70e81'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:55280 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55294 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:08:28,289 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0b8cfee8383e92b71d10 2025-12-11 20:08:28,290 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0b8cfee8383e92b71d10 HTTP/1.1" 502 150 2025-12-11 20:08:28,290 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0b8cfee8383e92b71d10'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:08:32,290 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0b8cfee8383e92b71d10 2025-12-11 20:08:32,295 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0b8cfee8383e92b71d10 HTTP/1.1" 502 150 2025-12-11 20:08:32,295 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0b8cfee8383e92b71d10'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:08:35,789 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af590fee8383e92b6b42e 2025-12-11 20:08:35,790 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af590fee8383e92b6b42e HTTP/1.1" 502 150 2025-12-11 20:08:35,791 - lava_callback - ERROR - Error processing callback for node 693af590fee8383e92b6b42e: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af590fee8383e92b6b42e (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af590fee8383e92b6b42e (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af590fee8383e92b6b42e (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:58054 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:08:37,365 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b16fb15f407b7629df285 2025-12-11 20:08:37,366 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b16fb15f407b7629df285 HTTP/1.1" 502 150 2025-12-11 20:08:37,367 - lava_callback - ERROR - Error processing callback for node 693b16fb15f407b7629df285: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b16fb15f407b7629df285 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b16fb15f407b7629df285 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b16fb15f407b7629df285 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:08:40,295 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0b8cfee8383e92b71d10 2025-12-11 20:08:40,296 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0b8cfee8383e92b71d10 HTTP/1.1" 502 150 2025-12-11 20:08:40,296 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0b8cfee8383e92b71d10'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:08:42,439 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b077efee8383e92b70e81 2025-12-11 20:08:42,440 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b077efee8383e92b70e81 HTTP/1.1" 502 150 2025-12-11 20:08:42,440 - lava_callback - ERROR - Error processing callback for node 693b077efee8383e92b70e81: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b077efee8383e92b70e81 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b077efee8383e92b70e81 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b077efee8383e92b70e81 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:60554 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:08:50,036 - lava_callback - INFO - Received callback for node 693b07acfee8383e92b70f4b INFO: 172.21.0.1:36008 - "POST /node/693b07acfee8383e92b70f4b HTTP/1.0" 202 Accepted 2025-12-11 20:08:50,045 - lava_callback - INFO - Starting async job submit for node 693b07acfee8383e92b70f4b 2025-12-11 20:08:50,108 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:08:50,122 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07acfee8383e92b70f4b HTTP/1.1" 502 150 2025-12-11 20:08:50,122 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07acfee8383e92b70f4b'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:08:50,122 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07acfee8383e92b70f4b 2025-12-11 20:08:50,122 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07acfee8383e92b70f4b HTTP/1.1" 502 150 2025-12-11 20:08:50,123 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07acfee8383e92b70f4b'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:08:52,123 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07acfee8383e92b70f4b 2025-12-11 20:08:52,124 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07acfee8383e92b70f4b HTTP/1.1" 502 150 2025-12-11 20:08:52,124 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07acfee8383e92b70f4b'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:08:55,023 - lava_callback - INFO - Received callback for node 693ae9b1fee8383e92b65dfb INFO: 172.21.0.1:36020 - "POST /node/693ae9b1fee8383e92b65dfb HTTP/1.0" 202 Accepted 2025-12-11 20:08:55,029 - lava_callback - INFO - Starting async job submit for node 693ae9b1fee8383e92b65dfb 2025-12-11 20:08:55,050 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:08:55,076 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9b1fee8383e92b65dfb HTTP/1.1" 502 150 2025-12-11 20:08:55,076 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9b1fee8383e92b65dfb'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:08:55,076 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9b1fee8383e92b65dfb 2025-12-11 20:08:55,076 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9b1fee8383e92b65dfb HTTP/1.1" 502 150 2025-12-11 20:08:55,076 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9b1fee8383e92b65dfb'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:08:56,124 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07acfee8383e92b70f4b 2025-12-11 20:08:56,125 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07acfee8383e92b70f4b HTTP/1.1" 502 150 2025-12-11 20:08:56,125 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07acfee8383e92b70f4b'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:08:56,297 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0b8cfee8383e92b71d10 2025-12-11 20:08:56,303 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0b8cfee8383e92b71d10 HTTP/1.1" 502 150 2025-12-11 20:08:56,303 - lava_callback - ERROR - Error processing callback for node 693b0b8cfee8383e92b71d10: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0b8cfee8383e92b71d10 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0b8cfee8383e92b71d10 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0b8cfee8383e92b71d10 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:36036 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:08:57,077 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9b1fee8383e92b65dfb 2025-12-11 20:08:57,078 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9b1fee8383e92b65dfb HTTP/1.1" 502 150 2025-12-11 20:08:57,078 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9b1fee8383e92b65dfb'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:09:01,079 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9b1fee8383e92b65dfb 2025-12-11 20:09:01,080 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9b1fee8383e92b65dfb HTTP/1.1" 502 150 2025-12-11 20:09:01,080 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9b1fee8383e92b65dfb'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:09:04,126 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07acfee8383e92b70f4b 2025-12-11 20:09:04,127 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07acfee8383e92b70f4b HTTP/1.1" 502 150 2025-12-11 20:09:04,127 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07acfee8383e92b70f4b'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:35960 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:09:09,080 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9b1fee8383e92b65dfb 2025-12-11 20:09:09,081 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9b1fee8383e92b65dfb HTTP/1.1" 502 150 2025-12-11 20:09:09,081 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9b1fee8383e92b65dfb'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:56178 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:56190 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:09:20,127 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07acfee8383e92b70f4b 2025-12-11 20:09:20,129 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07acfee8383e92b70f4b HTTP/1.1" 502 150 2025-12-11 20:09:20,130 - lava_callback - ERROR - Error processing callback for node 693b07acfee8383e92b70f4b: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b07acfee8383e92b70f4b (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b07acfee8383e92b70f4b (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b07acfee8383e92b70f4b (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:09:25,082 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9b1fee8383e92b65dfb 2025-12-11 20:09:25,083 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9b1fee8383e92b65dfb HTTP/1.1" 502 150 2025-12-11 20:09:25,083 - lava_callback - ERROR - Error processing callback for node 693ae9b1fee8383e92b65dfb: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9b1fee8383e92b65dfb (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9b1fee8383e92b65dfb (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9b1fee8383e92b65dfb (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:58464 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58478 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:09:29,324 - lava_callback - INFO - Received callback for node 693b07b2fee8383e92b70f67 INFO: 172.21.0.1:58494 - "POST /node/693b07b2fee8383e92b70f67 HTTP/1.0" 202 Accepted 2025-12-11 20:09:29,334 - lava_callback - INFO - Starting async job submit for node 693b07b2fee8383e92b70f67 2025-12-11 20:09:29,402 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:09:29,415 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07b2fee8383e92b70f67 HTTP/1.1" 502 150 2025-12-11 20:09:29,415 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07b2fee8383e92b70f67'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:09:29,415 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07b2fee8383e92b70f67 2025-12-11 20:09:29,416 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07b2fee8383e92b70f67 HTTP/1.1" 502 150 2025-12-11 20:09:29,416 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07b2fee8383e92b70f67'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:09:31,416 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07b2fee8383e92b70f67 2025-12-11 20:09:31,418 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07b2fee8383e92b70f67 HTTP/1.1" 502 150 2025-12-11 20:09:31,418 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07b2fee8383e92b70f67'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:09:35,418 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07b2fee8383e92b70f67 2025-12-11 20:09:35,419 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07b2fee8383e92b70f67 HTTP/1.1" 502 150 2025-12-11 20:09:35,419 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07b2fee8383e92b70f67'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:55026 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:09:37,718 - lava_callback - INFO - Received callback for node 693b1baf15f407b7629e02a4 INFO: 172.21.0.1:55034 - "POST /node/693b1baf15f407b7629e02a4 HTTP/1.0" 202 Accepted 2025-12-11 20:09:37,725 - lava_callback - INFO - Starting async job submit for node 693b1baf15f407b7629e02a4 2025-12-11 20:09:37,748 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:09:37,784 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1baf15f407b7629e02a4 HTTP/1.1" 502 150 2025-12-11 20:09:37,784 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1baf15f407b7629e02a4'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:09:37,784 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1baf15f407b7629e02a4 2025-12-11 20:09:37,785 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1baf15f407b7629e02a4 HTTP/1.1" 502 150 2025-12-11 20:09:37,785 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1baf15f407b7629e02a4'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:09:39,785 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1baf15f407b7629e02a4 2025-12-11 20:09:39,786 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1baf15f407b7629e02a4 HTTP/1.1" 502 150 2025-12-11 20:09:39,786 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1baf15f407b7629e02a4'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:09:43,419 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07b2fee8383e92b70f67 2025-12-11 20:09:43,420 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07b2fee8383e92b70f67 HTTP/1.1" 502 150 2025-12-11 20:09:43,420 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07b2fee8383e92b70f67'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:09:43,787 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1baf15f407b7629e02a4 2025-12-11 20:09:43,788 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1baf15f407b7629e02a4 HTTP/1.1" 502 150 2025-12-11 20:09:43,788 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1baf15f407b7629e02a4'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:09:46,263 - lava_callback - INFO - Received callback for node 693ae645fee8383e92b64c80 INFO: 172.21.0.1:57854 - "POST /node/693ae645fee8383e92b64c80 HTTP/1.0" 202 Accepted 2025-12-11 20:09:46,270 - lava_callback - INFO - Starting async job submit for node 693ae645fee8383e92b64c80 2025-12-11 20:09:46,328 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:09:46,338 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae645fee8383e92b64c80 HTTP/1.1" 502 150 2025-12-11 20:09:46,338 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae645fee8383e92b64c80'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:09:46,338 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae645fee8383e92b64c80 2025-12-11 20:09:46,339 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae645fee8383e92b64c80 HTTP/1.1" 502 150 2025-12-11 20:09:46,339 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae645fee8383e92b64c80'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:57870 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:09:48,339 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae645fee8383e92b64c80 2025-12-11 20:09:48,340 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae645fee8383e92b64c80 HTTP/1.1" 502 150 2025-12-11 20:09:48,340 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae645fee8383e92b64c80'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:09:51,788 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1baf15f407b7629e02a4 2025-12-11 20:09:51,790 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1baf15f407b7629e02a4 HTTP/1.1" 502 150 2025-12-11 20:09:51,790 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1baf15f407b7629e02a4'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:09:52,341 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae645fee8383e92b64c80 2025-12-11 20:09:52,342 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae645fee8383e92b64c80 HTTP/1.1" 502 150 2025-12-11 20:09:52,342 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae645fee8383e92b64c80'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:09:52,415 - lava_callback - INFO - Received callback for node 693b07c7fee8383e92b7101b INFO: 172.21.0.1:32892 - "POST /node/693b07c7fee8383e92b7101b HTTP/1.0" 202 Accepted 2025-12-11 20:09:52,424 - lava_callback - INFO - Starting async job submit for node 693b07c7fee8383e92b7101b 2025-12-11 20:09:52,486 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:09:52,496 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07c7fee8383e92b7101b HTTP/1.1" 502 150 2025-12-11 20:09:52,496 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07c7fee8383e92b7101b'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:09:52,496 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07c7fee8383e92b7101b 2025-12-11 20:09:52,497 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07c7fee8383e92b7101b HTTP/1.1" 502 150 2025-12-11 20:09:52,497 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07c7fee8383e92b7101b'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:09:54,497 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07c7fee8383e92b7101b 2025-12-11 20:09:54,498 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07c7fee8383e92b7101b HTTP/1.1" 502 150 2025-12-11 20:09:54,498 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07c7fee8383e92b7101b'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:32900 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:09:58,498 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07c7fee8383e92b7101b 2025-12-11 20:09:58,500 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07c7fee8383e92b7101b HTTP/1.1" 502 150 2025-12-11 20:09:58,500 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07c7fee8383e92b7101b'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:09:59,421 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07b2fee8383e92b70f67 2025-12-11 20:09:59,422 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07b2fee8383e92b70f67 HTTP/1.1" 502 150 2025-12-11 20:09:59,422 - lava_callback - ERROR - Error processing callback for node 693b07b2fee8383e92b70f67: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b07b2fee8383e92b70f67 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b07b2fee8383e92b70f67 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b07b2fee8383e92b70f67 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:10:00,342 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae645fee8383e92b64c80 2025-12-11 20:10:00,343 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae645fee8383e92b64c80 HTTP/1.1" 502 150 2025-12-11 20:10:00,343 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae645fee8383e92b64c80'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:10:06,500 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07c7fee8383e92b7101b 2025-12-11 20:10:06,501 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07c7fee8383e92b7101b HTTP/1.1" 502 150 2025-12-11 20:10:06,501 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07c7fee8383e92b7101b'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:41980 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:10:06,795 - lava_callback - INFO - Received callback for node 693ae865fee8383e92b658fc INFO: 172.21.0.1:41996 - "POST /node/693ae865fee8383e92b658fc HTTP/1.0" 202 Accepted 2025-12-11 20:10:06,802 - lava_callback - INFO - Starting async job submit for node 693ae865fee8383e92b658fc 2025-12-11 20:10:06,835 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:10:06,853 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae865fee8383e92b658fc HTTP/1.1" 502 150 2025-12-11 20:10:06,853 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae865fee8383e92b658fc'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:10:06,853 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae865fee8383e92b658fc 2025-12-11 20:10:06,854 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae865fee8383e92b658fc HTTP/1.1" 502 150 2025-12-11 20:10:06,854 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae865fee8383e92b658fc'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:10:07,790 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1baf15f407b7629e02a4 2025-12-11 20:10:07,791 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1baf15f407b7629e02a4 HTTP/1.1" 502 150 2025-12-11 20:10:07,791 - lava_callback - ERROR - Error processing callback for node 693b1baf15f407b7629e02a4: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1baf15f407b7629e02a4 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1baf15f407b7629e02a4 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1baf15f407b7629e02a4 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:10:08,854 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae865fee8383e92b658fc 2025-12-11 20:10:08,856 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae865fee8383e92b658fc HTTP/1.1" 502 150 2025-12-11 20:10:08,856 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae865fee8383e92b658fc'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:10:12,856 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae865fee8383e92b658fc 2025-12-11 20:10:12,857 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae865fee8383e92b658fc HTTP/1.1" 502 150 2025-12-11 20:10:12,858 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae865fee8383e92b658fc'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:10:13,383 - lava_callback - INFO - Received callback for node 693affacfee8383e92b6e549 INFO: 172.21.0.1:35812 - "POST /node/693affacfee8383e92b6e549 HTTP/1.0" 202 Accepted 2025-12-11 20:10:13,390 - lava_callback - INFO - Starting async job submit for node 693affacfee8383e92b6e549 2025-12-11 20:10:13,453 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:10:13,463 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693affacfee8383e92b6e549 HTTP/1.1" 502 150 2025-12-11 20:10:13,463 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693affacfee8383e92b6e549'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:10:13,463 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693affacfee8383e92b6e549 2025-12-11 20:10:13,464 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693affacfee8383e92b6e549 HTTP/1.1" 502 150 2025-12-11 20:10:13,464 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693affacfee8383e92b6e549'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:35828 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 20:10:15,464 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693affacfee8383e92b6e549 2025-12-11 20:10:15,466 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693affacfee8383e92b6e549 HTTP/1.1" 502 150 2025-12-11 20:10:15,466 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693affacfee8383e92b6e549'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:10:16,343 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae645fee8383e92b64c80 2025-12-11 20:10:16,345 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae645fee8383e92b64c80 HTTP/1.1" 502 150 2025-12-11 20:10:16,345 - lava_callback - ERROR - Error processing callback for node 693ae645fee8383e92b64c80: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae645fee8383e92b64c80 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae645fee8383e92b64c80 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae645fee8383e92b64c80 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:35832 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:10:19,466 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693affacfee8383e92b6e549 2025-12-11 20:10:19,468 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693affacfee8383e92b6e549 HTTP/1.1" 502 150 2025-12-11 20:10:19,468 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693affacfee8383e92b6e549'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:10:20,858 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae865fee8383e92b658fc 2025-12-11 20:10:20,859 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae865fee8383e92b658fc HTTP/1.1" 502 150 2025-12-11 20:10:20,859 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae865fee8383e92b658fc'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:10:22,501 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07c7fee8383e92b7101b 2025-12-11 20:10:22,503 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07c7fee8383e92b7101b HTTP/1.1" 502 150 2025-12-11 20:10:22,503 - lava_callback - ERROR - Error processing callback for node 693b07c7fee8383e92b7101b: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b07c7fee8383e92b7101b (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b07c7fee8383e92b7101b (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b07c7fee8383e92b7101b (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:10:25,985 - lava_callback - INFO - Received callback for node 693b0cebfee8383e92b720de INFO: 172.21.0.1:47318 - "POST /node/693b0cebfee8383e92b720de HTTP/1.0" 202 Accepted 2025-12-11 20:10:25,992 - lava_callback - INFO - Starting async job submit for node 693b0cebfee8383e92b720de 2025-12-11 20:10:26,015 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:10:26,025 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0cebfee8383e92b720de HTTP/1.1" 502 150 2025-12-11 20:10:26,025 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0cebfee8383e92b720de'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:10:26,025 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0cebfee8383e92b720de 2025-12-11 20:10:26,026 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0cebfee8383e92b720de HTTP/1.1" 502 150 2025-12-11 20:10:26,026 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0cebfee8383e92b720de'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:47320 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47326 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:10:27,468 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693affacfee8383e92b6e549 2025-12-11 20:10:27,469 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693affacfee8383e92b6e549 HTTP/1.1" 502 150 2025-12-11 20:10:27,469 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693affacfee8383e92b6e549'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:10:28,026 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0cebfee8383e92b720de 2025-12-11 20:10:28,027 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0cebfee8383e92b720de HTTP/1.1" 502 150 2025-12-11 20:10:28,028 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0cebfee8383e92b720de'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:10:32,028 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0cebfee8383e92b720de 2025-12-11 20:10:32,029 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0cebfee8383e92b720de HTTP/1.1" 502 150 2025-12-11 20:10:32,029 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0cebfee8383e92b720de'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:44050 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:10:36,859 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae865fee8383e92b658fc 2025-12-11 20:10:36,861 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae865fee8383e92b658fc HTTP/1.1" 502 150 2025-12-11 20:10:36,861 - lava_callback - ERROR - Error processing callback for node 693ae865fee8383e92b658fc: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae865fee8383e92b658fc (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae865fee8383e92b658fc (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae865fee8383e92b658fc (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:10:40,029 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0cebfee8383e92b720de 2025-12-11 20:10:40,030 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0cebfee8383e92b720de HTTP/1.1" 502 150 2025-12-11 20:10:40,031 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0cebfee8383e92b720de'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:10:43,470 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693affacfee8383e92b6e549 2025-12-11 20:10:43,471 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693affacfee8383e92b6e549 HTTP/1.1" 502 150 2025-12-11 20:10:43,471 - lava_callback - ERROR - Error processing callback for node 693affacfee8383e92b6e549: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693affacfee8383e92b6e549 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693affacfee8383e92b6e549 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693affacfee8383e92b6e549 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:51516 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:10:46,990 - lava_callback - INFO - Received callback for node 693ae861fee8383e92b658e2 INFO: 172.21.0.1:51528 - "POST /node/693ae861fee8383e92b658e2 HTTP/1.0" 202 Accepted 2025-12-11 20:10:46,996 - lava_callback - INFO - Starting async job submit for node 693ae861fee8383e92b658e2 2025-12-11 20:10:47,037 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:10:47,047 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae861fee8383e92b658e2 HTTP/1.1" 502 150 2025-12-11 20:10:47,047 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae861fee8383e92b658e2'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:10:47,047 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae861fee8383e92b658e2 2025-12-11 20:10:47,047 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae861fee8383e92b658e2 HTTP/1.1" 502 150 2025-12-11 20:10:47,047 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae861fee8383e92b658e2'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:10:49,048 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae861fee8383e92b658e2 2025-12-11 20:10:49,049 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae861fee8383e92b658e2 HTTP/1.1" 502 150 2025-12-11 20:10:49,049 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae861fee8383e92b658e2'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:10:50,161 - lava_callback - INFO - Received callback for node 693aedfefee8383e92b67a72 INFO: 172.21.0.1:39996 - "POST /node/693aedfefee8383e92b67a72 HTTP/1.0" 202 Accepted 2025-12-11 20:10:50,168 - lava_callback - INFO - Starting async job submit for node 693aedfefee8383e92b67a72 2025-12-11 20:10:50,202 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:10:50,219 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aedfefee8383e92b67a72 HTTP/1.1" 502 150 2025-12-11 20:10:50,220 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aedfefee8383e92b67a72'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:10:50,220 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aedfefee8383e92b67a72 2025-12-11 20:10:50,220 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aedfefee8383e92b67a72 HTTP/1.1" 502 150 2025-12-11 20:10:50,220 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aedfefee8383e92b67a72'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:10:52,221 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aedfefee8383e92b67a72 2025-12-11 20:10:52,222 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aedfefee8383e92b67a72 HTTP/1.1" 502 150 2025-12-11 20:10:52,222 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aedfefee8383e92b67a72'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:10:53,049 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae861fee8383e92b658e2 2025-12-11 20:10:53,050 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae861fee8383e92b658e2 HTTP/1.1" 502 150 2025-12-11 20:10:53,050 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae861fee8383e92b658e2'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:10:56,031 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0cebfee8383e92b720de 2025-12-11 20:10:56,032 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0cebfee8383e92b720de HTTP/1.1" 502 150 2025-12-11 20:10:56,032 - lava_callback - ERROR - Error processing callback for node 693b0cebfee8383e92b720de: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0cebfee8383e92b720de (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0cebfee8383e92b720de (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0cebfee8383e92b720de (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:10:56,222 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aedfefee8383e92b67a72 2025-12-11 20:10:56,223 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aedfefee8383e92b67a72 HTTP/1.1" 502 150 2025-12-11 20:10:56,223 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aedfefee8383e92b67a72'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:40004 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:10:59,862 - lava_callback - INFO - Received callback for node 693affa6fee8383e92b6e53a INFO: 172.21.0.1:33284 - "POST /node/693affa6fee8383e92b6e53a HTTP/1.0" 202 Accepted 2025-12-11 20:10:59,874 - lava_callback - INFO - Starting async job submit for node 693affa6fee8383e92b6e53a 2025-12-11 20:10:59,939 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:10:59,949 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693affa6fee8383e92b6e53a HTTP/1.1" 502 150 2025-12-11 20:10:59,949 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693affa6fee8383e92b6e53a'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:10:59,949 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693affa6fee8383e92b6e53a 2025-12-11 20:10:59,950 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693affa6fee8383e92b6e53a HTTP/1.1" 502 150 2025-12-11 20:10:59,950 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693affa6fee8383e92b6e53a'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:01,051 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae861fee8383e92b658e2 2025-12-11 20:11:01,052 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae861fee8383e92b658e2 HTTP/1.1" 502 150 2025-12-11 20:11:01,052 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae861fee8383e92b658e2'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:01,950 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693affa6fee8383e92b6e53a 2025-12-11 20:11:01,952 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693affa6fee8383e92b6e53a HTTP/1.1" 502 150 2025-12-11 20:11:01,952 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693affa6fee8383e92b6e53a'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:04,223 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aedfefee8383e92b67a72 2025-12-11 20:11:04,230 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aedfefee8383e92b67a72 HTTP/1.1" 502 150 2025-12-11 20:11:04,230 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aedfefee8383e92b67a72'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:05,886 - lava_callback - INFO - Received callback for node 693af5d5fee8383e92b6b587 INFO: 172.21.0.1:33292 - "POST /node/693af5d5fee8383e92b6b587 HTTP/1.0" 202 Accepted 2025-12-11 20:11:05,892 - lava_callback - INFO - Starting async job submit for node 693af5d5fee8383e92b6b587 2025-12-11 20:11:05,910 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:11:05,920 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af5d5fee8383e92b6b587 HTTP/1.1" 502 150 2025-12-11 20:11:05,920 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af5d5fee8383e92b6b587'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:05,920 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af5d5fee8383e92b6b587 2025-12-11 20:11:05,920 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af5d5fee8383e92b6b587 HTTP/1.1" 502 150 2025-12-11 20:11:05,920 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af5d5fee8383e92b6b587'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:05,952 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693affa6fee8383e92b6e53a 2025-12-11 20:11:05,953 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693affa6fee8383e92b6e53a HTTP/1.1" 502 150 2025-12-11 20:11:05,953 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693affa6fee8383e92b6e53a'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:33306 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:11:06,778 - lava_callback - INFO - Received callback for node 693ae860fee8383e92b658dd INFO: 172.21.0.1:33316 - "POST /node/693ae860fee8383e92b658dd HTTP/1.0" 202 Accepted 2025-12-11 20:11:06,785 - lava_callback - INFO - Starting async job submit for node 693ae860fee8383e92b658dd 2025-12-11 20:11:06,824 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:11:06,834 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae860fee8383e92b658dd HTTP/1.1" 502 150 2025-12-11 20:11:06,834 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae860fee8383e92b658dd'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:06,834 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae860fee8383e92b658dd 2025-12-11 20:11:06,835 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae860fee8383e92b658dd HTTP/1.1" 502 150 2025-12-11 20:11:06,835 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae860fee8383e92b658dd'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:07,475 - lava_callback - INFO - Received callback for node 693b12cdfee8383e92b73e7b INFO: 172.21.0.1:33322 - "POST /node/693b12cdfee8383e92b73e7b HTTP/1.0" 202 Accepted 2025-12-11 20:11:07,483 - lava_callback - INFO - Starting async job submit for node 693b12cdfee8383e92b73e7b 2025-12-11 20:11:07,518 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:11:07,549 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b12cdfee8383e92b73e7b HTTP/1.1" 502 150 2025-12-11 20:11:07,549 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b12cdfee8383e92b73e7b'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:07,549 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b12cdfee8383e92b73e7b 2025-12-11 20:11:07,550 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b12cdfee8383e92b73e7b HTTP/1.1" 502 150 2025-12-11 20:11:07,550 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b12cdfee8383e92b73e7b'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:07,921 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af5d5fee8383e92b6b587 2025-12-11 20:11:07,926 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af5d5fee8383e92b6b587 HTTP/1.1" 502 150 2025-12-11 20:11:07,926 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af5d5fee8383e92b6b587'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:08,835 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae860fee8383e92b658dd 2025-12-11 20:11:08,836 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae860fee8383e92b658dd HTTP/1.1" 502 150 2025-12-11 20:11:08,837 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae860fee8383e92b658dd'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:09,550 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b12cdfee8383e92b73e7b 2025-12-11 20:11:09,551 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b12cdfee8383e92b73e7b HTTP/1.1" 502 150 2025-12-11 20:11:09,551 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b12cdfee8383e92b73e7b'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:11,927 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af5d5fee8383e92b6b587 2025-12-11 20:11:11,928 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af5d5fee8383e92b6b587 HTTP/1.1" 502 150 2025-12-11 20:11:11,928 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af5d5fee8383e92b6b587'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:12,725 - lava_callback - INFO - Received callback for node 693b09f5fee8383e92b7192a INFO: 172.21.0.1:58494 - "POST /node/693b09f5fee8383e92b7192a HTTP/1.0" 202 Accepted 2025-12-11 20:11:12,732 - lava_callback - INFO - Starting async job submit for node 693b09f5fee8383e92b7192a 2025-12-11 20:11:12,753 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:11:12,770 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b09f5fee8383e92b7192a HTTP/1.1" 502 150 2025-12-11 20:11:12,770 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b09f5fee8383e92b7192a'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:12,770 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b09f5fee8383e92b7192a 2025-12-11 20:11:12,771 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b09f5fee8383e92b7192a HTTP/1.1" 502 150 2025-12-11 20:11:12,771 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b09f5fee8383e92b7192a'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:12,837 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae860fee8383e92b658dd 2025-12-11 20:11:12,838 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae860fee8383e92b658dd HTTP/1.1" 502 150 2025-12-11 20:11:12,838 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae860fee8383e92b658dd'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:13,552 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b12cdfee8383e92b73e7b 2025-12-11 20:11:13,553 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b12cdfee8383e92b73e7b HTTP/1.1" 502 150 2025-12-11 20:11:13,553 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b12cdfee8383e92b73e7b'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:13,953 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693affa6fee8383e92b6e53a 2025-12-11 20:11:13,958 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693affa6fee8383e92b6e53a HTTP/1.1" 502 150 2025-12-11 20:11:13,958 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693affa6fee8383e92b6e53a'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:14,771 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b09f5fee8383e92b7192a 2025-12-11 20:11:14,772 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b09f5fee8383e92b7192a HTTP/1.1" 502 150 2025-12-11 20:11:14,772 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b09f5fee8383e92b7192a'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:58498 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:58514 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:11:17,052 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae861fee8383e92b658e2 2025-12-11 20:11:17,054 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae861fee8383e92b658e2 HTTP/1.1" 502 150 2025-12-11 20:11:17,054 - lava_callback - ERROR - Error processing callback for node 693ae861fee8383e92b658e2: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae861fee8383e92b658e2 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae861fee8383e92b658e2 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae861fee8383e92b658e2 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:11:18,772 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b09f5fee8383e92b7192a 2025-12-11 20:11:18,774 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b09f5fee8383e92b7192a HTTP/1.1" 502 150 2025-12-11 20:11:18,774 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b09f5fee8383e92b7192a'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:19,928 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af5d5fee8383e92b6b587 2025-12-11 20:11:19,930 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af5d5fee8383e92b6b587 HTTP/1.1" 502 150 2025-12-11 20:11:19,930 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af5d5fee8383e92b6b587'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:20,230 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aedfefee8383e92b67a72 2025-12-11 20:11:20,231 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aedfefee8383e92b67a72 HTTP/1.1" 502 150 2025-12-11 20:11:20,232 - lava_callback - ERROR - Error processing callback for node 693aedfefee8383e92b67a72: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aedfefee8383e92b67a72 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aedfefee8383e92b67a72 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aedfefee8383e92b67a72 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:11:20,838 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae860fee8383e92b658dd 2025-12-11 20:11:20,839 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae860fee8383e92b658dd HTTP/1.1" 502 150 2025-12-11 20:11:20,839 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae860fee8383e92b658dd'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:21,553 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b12cdfee8383e92b73e7b 2025-12-11 20:11:21,554 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b12cdfee8383e92b73e7b HTTP/1.1" 502 150 2025-12-11 20:11:21,555 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b12cdfee8383e92b73e7b'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:40028 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:11:26,774 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b09f5fee8383e92b7192a 2025-12-11 20:11:26,775 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b09f5fee8383e92b7192a HTTP/1.1" 502 150 2025-12-11 20:11:26,775 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b09f5fee8383e92b7192a'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:26,995 - lava_callback - INFO - Received callback for node 693ae86afee8383e92b65917 INFO: 172.21.0.1:40030 - "POST /node/693ae86afee8383e92b65917 HTTP/1.0" 202 Accepted 2025-12-11 20:11:27,002 - lava_callback - INFO - Starting async job submit for node 693ae86afee8383e92b65917 2025-12-11 20:11:27,037 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:11:27,070 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae86afee8383e92b65917 HTTP/1.1" 502 150 2025-12-11 20:11:27,070 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae86afee8383e92b65917'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:27,070 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae86afee8383e92b65917 2025-12-11 20:11:27,071 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae86afee8383e92b65917 HTTP/1.1" 502 150 2025-12-11 20:11:27,071 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae86afee8383e92b65917'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:40034 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:11:29,071 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae86afee8383e92b65917 2025-12-11 20:11:29,073 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae86afee8383e92b65917 HTTP/1.1" 502 150 2025-12-11 20:11:29,074 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae86afee8383e92b65917'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:29,958 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693affa6fee8383e92b6e53a 2025-12-11 20:11:29,959 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693affa6fee8383e92b6e53a HTTP/1.1" 502 150 2025-12-11 20:11:29,959 - lava_callback - ERROR - Error processing callback for node 693affa6fee8383e92b6e53a: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693affa6fee8383e92b6e53a (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693affa6fee8383e92b6e53a (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693affa6fee8383e92b6e53a (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:11:33,074 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae86afee8383e92b65917 2025-12-11 20:11:33,075 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae86afee8383e92b65917 HTTP/1.1" 502 150 2025-12-11 20:11:33,075 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae86afee8383e92b65917'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:35,930 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af5d5fee8383e92b6b587 2025-12-11 20:11:35,931 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af5d5fee8383e92b6b587 HTTP/1.1" 502 150 2025-12-11 20:11:35,931 - lava_callback - ERROR - Error processing callback for node 693af5d5fee8383e92b6b587: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af5d5fee8383e92b6b587 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af5d5fee8383e92b6b587 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af5d5fee8383e92b6b587 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:35042 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:11:36,840 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae860fee8383e92b658dd 2025-12-11 20:11:36,841 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae860fee8383e92b658dd HTTP/1.1" 502 150 2025-12-11 20:11:36,841 - lava_callback - ERROR - Error processing callback for node 693ae860fee8383e92b658dd: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae860fee8383e92b658dd (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae860fee8383e92b658dd (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae860fee8383e92b658dd (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:11:37,555 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b12cdfee8383e92b73e7b 2025-12-11 20:11:37,556 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b12cdfee8383e92b73e7b HTTP/1.1" 502 150 2025-12-11 20:11:37,556 - lava_callback - ERROR - Error processing callback for node 693b12cdfee8383e92b73e7b: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b12cdfee8383e92b73e7b (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b12cdfee8383e92b73e7b (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b12cdfee8383e92b73e7b (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:11:37,592 - lava_callback - INFO - Received callback for node 693b1bb415f407b7629e02ac INFO: 172.21.0.1:35044 - "POST /node/693b1bb415f407b7629e02ac HTTP/1.0" 202 Accepted 2025-12-11 20:11:37,598 - lava_callback - INFO - Starting async job submit for node 693b1bb415f407b7629e02ac 2025-12-11 20:11:37,619 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:11:37,640 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bb415f407b7629e02ac HTTP/1.1" 502 150 2025-12-11 20:11:37,640 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bb415f407b7629e02ac'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:37,640 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bb415f407b7629e02ac 2025-12-11 20:11:37,641 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bb415f407b7629e02ac HTTP/1.1" 502 150 2025-12-11 20:11:37,641 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bb415f407b7629e02ac'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:39,641 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bb415f407b7629e02ac 2025-12-11 20:11:39,642 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bb415f407b7629e02ac HTTP/1.1" 502 150 2025-12-11 20:11:39,643 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bb415f407b7629e02ac'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:41,075 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae86afee8383e92b65917 2025-12-11 20:11:41,077 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae86afee8383e92b65917 HTTP/1.1" 502 150 2025-12-11 20:11:41,077 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae86afee8383e92b65917'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:42,775 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b09f5fee8383e92b7192a 2025-12-11 20:11:42,777 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b09f5fee8383e92b7192a HTTP/1.1" 502 150 2025-12-11 20:11:42,777 - lava_callback - ERROR - Error processing callback for node 693b09f5fee8383e92b7192a: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b09f5fee8383e92b7192a (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b09f5fee8383e92b7192a (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b09f5fee8383e92b7192a (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:11:43,643 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bb415f407b7629e02ac 2025-12-11 20:11:43,644 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bb415f407b7629e02ac HTTP/1.1" 502 150 2025-12-11 20:11:43,644 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bb415f407b7629e02ac'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:46,092 - lava_callback - INFO - Received callback for node 693b0dbffee8383e92b722e4 INFO: 172.21.0.1:35376 - "POST /node/693b0dbffee8383e92b722e4 HTTP/1.0" 202 Accepted 2025-12-11 20:11:46,102 - lava_callback - INFO - Starting async job submit for node 693b0dbffee8383e92b722e4 2025-12-11 20:11:46,127 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:11:46,147 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0dbffee8383e92b722e4 HTTP/1.1" 502 150 2025-12-11 20:11:46,147 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0dbffee8383e92b722e4'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:46,147 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0dbffee8383e92b722e4 2025-12-11 20:11:46,148 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0dbffee8383e92b722e4 HTTP/1.1" 502 150 2025-12-11 20:11:46,148 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0dbffee8383e92b722e4'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:35388 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:11:48,148 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0dbffee8383e92b722e4 2025-12-11 20:11:48,149 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0dbffee8383e92b722e4 HTTP/1.1" 502 150 2025-12-11 20:11:48,149 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0dbffee8383e92b722e4'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:51,644 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bb415f407b7629e02ac 2025-12-11 20:11:51,645 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bb415f407b7629e02ac HTTP/1.1" 502 150 2025-12-11 20:11:51,645 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bb415f407b7629e02ac'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:11:52,149 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0dbffee8383e92b722e4 2025-12-11 20:11:52,150 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0dbffee8383e92b722e4 HTTP/1.1" 502 150 2025-12-11 20:11:52,150 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0dbffee8383e92b722e4'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:47960 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:11:57,077 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae86afee8383e92b65917 2025-12-11 20:11:57,078 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae86afee8383e92b65917 HTTP/1.1" 502 150 2025-12-11 20:11:57,078 - lava_callback - ERROR - Error processing callback for node 693ae86afee8383e92b65917: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae86afee8383e92b65917 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae86afee8383e92b65917 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae86afee8383e92b65917 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:12:00,151 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0dbffee8383e92b722e4 2025-12-11 20:12:00,152 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0dbffee8383e92b722e4 HTTP/1.1" 502 150 2025-12-11 20:12:00,152 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0dbffee8383e92b722e4'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:41282 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:12:07,645 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bb415f407b7629e02ac 2025-12-11 20:12:07,647 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bb415f407b7629e02ac HTTP/1.1" 502 150 2025-12-11 20:12:07,647 - lava_callback - ERROR - Error processing callback for node 693b1bb415f407b7629e02ac: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1bb415f407b7629e02ac (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1bb415f407b7629e02ac (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1bb415f407b7629e02ac (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:12:10,400 - lava_callback - INFO - Received callback for node 693b07d9fee8383e92b71078 INFO: 172.21.0.1:47324 - "POST /node/693b07d9fee8383e92b71078 HTTP/1.0" 202 Accepted 2025-12-11 20:12:10,409 - lava_callback - INFO - Starting async job submit for node 693b07d9fee8383e92b71078 2025-12-11 20:12:10,494 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:12:10,494 - lava_callback - INFO - Received callback for node 693b1c1a15f407b7629e0358 INFO: 172.21.0.1:47330 - "POST /node/693b1c1a15f407b7629e0358 HTTP/1.0" 202 Accepted 2025-12-11 20:12:10,502 - lava_callback - INFO - Starting async job submit for node 693b1c1a15f407b7629e0358 2025-12-11 20:12:10,546 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:12:10,547 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07d9fee8383e92b71078 HTTP/1.1" 502 150 2025-12-11 20:12:10,547 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07d9fee8383e92b71078'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:12:10,547 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07d9fee8383e92b71078 2025-12-11 20:12:10,548 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07d9fee8383e92b71078 HTTP/1.1" 502 150 2025-12-11 20:12:10,548 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07d9fee8383e92b71078'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:12:10,557 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c1a15f407b7629e0358 HTTP/1.1" 502 150 2025-12-11 20:12:10,557 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c1a15f407b7629e0358'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:12:10,557 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c1a15f407b7629e0358 2025-12-11 20:12:10,558 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c1a15f407b7629e0358 HTTP/1.1" 502 150 2025-12-11 20:12:10,558 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c1a15f407b7629e0358'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:12:12,548 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07d9fee8383e92b71078 2025-12-11 20:12:12,550 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07d9fee8383e92b71078 HTTP/1.1" 502 150 2025-12-11 20:12:12,550 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07d9fee8383e92b71078'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:12:12,558 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c1a15f407b7629e0358 2025-12-11 20:12:12,559 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c1a15f407b7629e0358 HTTP/1.1" 502 150 2025-12-11 20:12:12,559 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c1a15f407b7629e0358'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:47332 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 20:12:16,152 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0dbffee8383e92b722e4 2025-12-11 20:12:16,153 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0dbffee8383e92b722e4 HTTP/1.1" 502 150 2025-12-11 20:12:16,153 - lava_callback - ERROR - Error processing callback for node 693b0dbffee8383e92b722e4: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0dbffee8383e92b722e4 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0dbffee8383e92b722e4 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0dbffee8383e92b722e4 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:47344 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:12:16,550 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07d9fee8383e92b71078 2025-12-11 20:12:16,552 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07d9fee8383e92b71078 HTTP/1.1" 502 150 2025-12-11 20:12:16,552 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07d9fee8383e92b71078'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:12:16,561 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c1a15f407b7629e0358 2025-12-11 20:12:16,562 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c1a15f407b7629e0358 HTTP/1.1" 502 150 2025-12-11 20:12:16,562 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c1a15f407b7629e0358'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:12:24,553 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07d9fee8383e92b71078 2025-12-11 20:12:24,554 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07d9fee8383e92b71078 HTTP/1.1" 502 150 2025-12-11 20:12:24,554 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b07d9fee8383e92b71078'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:12:24,562 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c1a15f407b7629e0358 2025-12-11 20:12:24,563 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c1a15f407b7629e0358 HTTP/1.1" 502 150 2025-12-11 20:12:24,563 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c1a15f407b7629e0358'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:12:25,831 - lava_callback - INFO - Received callback for node 693b16fb15f407b7629df287 INFO: 172.21.0.1:41012 - "POST /node/693b16fb15f407b7629df287 HTTP/1.0" 202 Accepted 2025-12-11 20:12:25,840 - lava_callback - INFO - Starting async job submit for node 693b16fb15f407b7629df287 2025-12-11 20:12:25,876 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:12:25,887 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b16fb15f407b7629df287 HTTP/1.1" 502 150 2025-12-11 20:12:25,887 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b16fb15f407b7629df287'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:12:25,888 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b16fb15f407b7629df287 2025-12-11 20:12:25,888 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b16fb15f407b7629df287 HTTP/1.1" 502 150 2025-12-11 20:12:25,888 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b16fb15f407b7629df287'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:41026 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41038 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:12:27,889 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b16fb15f407b7629df287 2025-12-11 20:12:27,890 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b16fb15f407b7629df287 HTTP/1.1" 502 150 2025-12-11 20:12:27,890 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b16fb15f407b7629df287'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:12:29,190 - lava_callback - INFO - Received callback for node 693aee31fee8383e92b67c75 INFO: 172.21.0.1:41040 - "POST /node/693aee31fee8383e92b67c75 HTTP/1.0" 202 Accepted 2025-12-11 20:12:29,197 - lava_callback - INFO - Starting async job submit for node 693aee31fee8383e92b67c75 2025-12-11 20:12:29,234 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:12:29,246 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aee31fee8383e92b67c75 HTTP/1.1" 502 150 2025-12-11 20:12:29,246 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aee31fee8383e92b67c75'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:12:29,247 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aee31fee8383e92b67c75 2025-12-11 20:12:29,247 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aee31fee8383e92b67c75 HTTP/1.1" 502 150 2025-12-11 20:12:29,247 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aee31fee8383e92b67c75'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:12:31,248 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aee31fee8383e92b67c75 2025-12-11 20:12:31,249 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aee31fee8383e92b67c75 HTTP/1.1" 502 150 2025-12-11 20:12:31,249 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aee31fee8383e92b67c75'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:12:31,890 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b16fb15f407b7629df287 2025-12-11 20:12:31,891 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b16fb15f407b7629df287 HTTP/1.1" 502 150 2025-12-11 20:12:31,891 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b16fb15f407b7629df287'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:12:35,249 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aee31fee8383e92b67c75 2025-12-11 20:12:35,250 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aee31fee8383e92b67c75 HTTP/1.1" 502 150 2025-12-11 20:12:35,250 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aee31fee8383e92b67c75'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:36848 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:12:39,892 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b16fb15f407b7629df287 2025-12-11 20:12:39,893 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b16fb15f407b7629df287 HTTP/1.1" 502 150 2025-12-11 20:12:39,893 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b16fb15f407b7629df287'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:12:40,554 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b07d9fee8383e92b71078 2025-12-11 20:12:40,556 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b07d9fee8383e92b71078 HTTP/1.1" 502 150 2025-12-11 20:12:40,556 - lava_callback - ERROR - Error processing callback for node 693b07d9fee8383e92b71078: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b07d9fee8383e92b71078 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b07d9fee8383e92b71078 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b07d9fee8383e92b71078 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:12:40,563 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c1a15f407b7629e0358 2025-12-11 20:12:40,564 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c1a15f407b7629e0358 HTTP/1.1" 502 150 2025-12-11 20:12:40,564 - lava_callback - ERROR - Error processing callback for node 693b1c1a15f407b7629e0358: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1c1a15f407b7629e0358 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1c1a15f407b7629e0358 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1c1a15f407b7629e0358 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:12:43,035 - lava_callback - INFO - Received callback for node 693b0a7bfee8383e92b71a80 INFO: 172.21.0.1:55286 - "POST /node/693b0a7bfee8383e92b71a80 HTTP/1.0" 202 Accepted 2025-12-11 20:12:43,044 - lava_callback - INFO - Starting async job submit for node 693b0a7bfee8383e92b71a80 2025-12-11 20:12:43,099 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:12:43,109 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0a7bfee8383e92b71a80 HTTP/1.1" 502 150 2025-12-11 20:12:43,109 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0a7bfee8383e92b71a80'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:12:43,109 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0a7bfee8383e92b71a80 2025-12-11 20:12:43,110 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0a7bfee8383e92b71a80 HTTP/1.1" 502 150 2025-12-11 20:12:43,110 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0a7bfee8383e92b71a80'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:12:43,250 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aee31fee8383e92b67c75 2025-12-11 20:12:43,251 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aee31fee8383e92b67c75 HTTP/1.1" 502 150 2025-12-11 20:12:43,252 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aee31fee8383e92b67c75'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:12:45,110 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0a7bfee8383e92b71a80 2025-12-11 20:12:45,111 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0a7bfee8383e92b71a80 HTTP/1.1" 502 150 2025-12-11 20:12:45,111 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0a7bfee8383e92b71a80'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:12:46,206 - lava_callback - INFO - Received callback for node 693ae866fee8383e92b65909 INFO: 172.21.0.1:55290 - "POST /node/693ae866fee8383e92b65909 HTTP/1.0" 202 Accepted 2025-12-11 20:12:46,213 - lava_callback - INFO - Starting async job submit for node 693ae866fee8383e92b65909 2025-12-11 20:12:46,245 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:12:46,255 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae866fee8383e92b65909 HTTP/1.1" 502 150 2025-12-11 20:12:46,255 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae866fee8383e92b65909'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:12:46,256 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae866fee8383e92b65909 2025-12-11 20:12:46,256 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae866fee8383e92b65909 HTTP/1.1" 502 150 2025-12-11 20:12:46,256 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae866fee8383e92b65909'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:55302 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:12:48,256 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae866fee8383e92b65909 2025-12-11 20:12:48,258 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae866fee8383e92b65909 HTTP/1.1" 502 150 2025-12-11 20:12:48,258 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae866fee8383e92b65909'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:12:49,112 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0a7bfee8383e92b71a80 2025-12-11 20:12:49,113 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0a7bfee8383e92b71a80 HTTP/1.1" 502 150 2025-12-11 20:12:49,113 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0a7bfee8383e92b71a80'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:12:52,258 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae866fee8383e92b65909 2025-12-11 20:12:52,259 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae866fee8383e92b65909 HTTP/1.1" 502 150 2025-12-11 20:12:52,259 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae866fee8383e92b65909'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:12:55,893 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b16fb15f407b7629df287 2025-12-11 20:12:55,894 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b16fb15f407b7629df287 HTTP/1.1" 502 150 2025-12-11 20:12:55,894 - lava_callback - ERROR - Error processing callback for node 693b16fb15f407b7629df287: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b16fb15f407b7629df287 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b16fb15f407b7629df287 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b16fb15f407b7629df287 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:32990 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:12:57,113 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0a7bfee8383e92b71a80 2025-12-11 20:12:57,114 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0a7bfee8383e92b71a80 HTTP/1.1" 502 150 2025-12-11 20:12:57,114 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0a7bfee8383e92b71a80'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:12:59,252 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aee31fee8383e92b67c75 2025-12-11 20:12:59,253 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aee31fee8383e92b67c75 HTTP/1.1" 502 150 2025-12-11 20:12:59,253 - lava_callback - ERROR - Error processing callback for node 693aee31fee8383e92b67c75: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aee31fee8383e92b67c75 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aee31fee8383e92b67c75 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aee31fee8383e92b67c75 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:13:00,260 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae866fee8383e92b65909 2025-12-11 20:13:00,261 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae866fee8383e92b65909 HTTP/1.1" 502 150 2025-12-11 20:13:00,261 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae866fee8383e92b65909'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:13:06,067 - lava_callback - INFO - Received callback for node 693af5eafee8383e92b6b5e5 INFO: 172.21.0.1:39048 - "POST /node/693af5eafee8383e92b6b5e5 HTTP/1.0" 202 Accepted 2025-12-11 20:13:06,072 - lava_callback - INFO - Starting async job submit for node 693af5eafee8383e92b6b5e5 2025-12-11 20:13:06,090 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:13:06,100 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af5eafee8383e92b6b5e5 HTTP/1.1" 502 150 2025-12-11 20:13:06,100 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af5eafee8383e92b6b5e5'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:13:06,100 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af5eafee8383e92b6b5e5 2025-12-11 20:13:06,101 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af5eafee8383e92b6b5e5 HTTP/1.1" 502 150 2025-12-11 20:13:06,101 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af5eafee8383e92b6b5e5'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:39062 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:13:07,562 - lava_callback - INFO - Received callback for node 693b0e13fee8383e92b72486 INFO: 172.21.0.1:39066 - "POST /node/693b0e13fee8383e92b72486 HTTP/1.0" 202 Accepted 2025-12-11 20:13:07,572 - lava_callback - INFO - Starting async job submit for node 693b0e13fee8383e92b72486 2025-12-11 20:13:07,604 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:13:07,617 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e13fee8383e92b72486 HTTP/1.1" 502 150 2025-12-11 20:13:07,617 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e13fee8383e92b72486'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:13:07,617 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e13fee8383e92b72486 2025-12-11 20:13:07,618 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e13fee8383e92b72486 HTTP/1.1" 502 150 2025-12-11 20:13:07,618 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e13fee8383e92b72486'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:13:08,102 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af5eafee8383e92b6b5e5 2025-12-11 20:13:08,106 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af5eafee8383e92b6b5e5 HTTP/1.1" 502 150 2025-12-11 20:13:08,106 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af5eafee8383e92b6b5e5'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:13:09,618 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e13fee8383e92b72486 2025-12-11 20:13:09,619 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e13fee8383e92b72486 HTTP/1.1" 502 150 2025-12-11 20:13:09,619 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e13fee8383e92b72486'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:13:12,106 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af5eafee8383e92b6b5e5 2025-12-11 20:13:12,107 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af5eafee8383e92b6b5e5 HTTP/1.1" 502 150 2025-12-11 20:13:12,107 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af5eafee8383e92b6b5e5'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:13:13,114 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0a7bfee8383e92b71a80 2025-12-11 20:13:13,116 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0a7bfee8383e92b71a80 HTTP/1.1" 502 150 2025-12-11 20:13:13,116 - lava_callback - ERROR - Error processing callback for node 693b0a7bfee8383e92b71a80: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0a7bfee8383e92b71a80 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0a7bfee8383e92b71a80 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0a7bfee8383e92b71a80 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:13:13,620 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e13fee8383e92b72486 2025-12-11 20:13:13,621 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e13fee8383e92b72486 HTTP/1.1" 502 150 2025-12-11 20:13:13,621 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e13fee8383e92b72486'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:47142 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 20:13:16,261 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae866fee8383e92b65909 2025-12-11 20:13:16,263 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae866fee8383e92b65909 HTTP/1.1" 502 150 2025-12-11 20:13:16,263 - lava_callback - ERROR - Error processing callback for node 693ae866fee8383e92b65909: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae866fee8383e92b65909 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae866fee8383e92b65909 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae866fee8383e92b65909 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:47152 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:13:20,108 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af5eafee8383e92b6b5e5 2025-12-11 20:13:20,109 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af5eafee8383e92b6b5e5 HTTP/1.1" 502 150 2025-12-11 20:13:20,109 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af5eafee8383e92b6b5e5'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:13:21,621 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e13fee8383e92b72486 2025-12-11 20:13:21,622 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e13fee8383e92b72486 HTTP/1.1" 502 150 2025-12-11 20:13:21,622 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e13fee8383e92b72486'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:35688 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35700 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:13:36,109 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af5eafee8383e92b6b5e5 2025-12-11 20:13:36,111 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af5eafee8383e92b6b5e5 HTTP/1.1" 502 150 2025-12-11 20:13:36,111 - lava_callback - ERROR - Error processing callback for node 693af5eafee8383e92b6b5e5: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af5eafee8383e92b6b5e5 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af5eafee8383e92b6b5e5 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af5eafee8383e92b6b5e5 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:49150 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:13:37,623 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e13fee8383e92b72486 2025-12-11 20:13:37,624 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e13fee8383e92b72486 HTTP/1.1" 502 150 2025-12-11 20:13:37,624 - lava_callback - ERROR - Error processing callback for node 693b0e13fee8383e92b72486: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e13fee8383e92b72486 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e13fee8383e92b72486 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e13fee8383e92b72486 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:13:46,253 - lava_callback - INFO - Received callback for node 693b1406fee8383e92b742d0 INFO: 172.21.0.1:43486 - "POST /node/693b1406fee8383e92b742d0 HTTP/1.0" 202 Accepted 2025-12-11 20:13:46,259 - lava_callback - INFO - Starting async job submit for node 693b1406fee8383e92b742d0 2025-12-11 20:13:46,282 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:13:46,320 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1406fee8383e92b742d0 HTTP/1.1" 502 150 2025-12-11 20:13:46,320 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1406fee8383e92b742d0'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:13:46,320 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1406fee8383e92b742d0 2025-12-11 20:13:46,321 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1406fee8383e92b742d0 HTTP/1.1" 502 150 2025-12-11 20:13:46,321 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1406fee8383e92b742d0'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:43492 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:13:47,705 - lava_callback - INFO - Received callback for node 693b16fb15f407b7629df289 INFO: 172.21.0.1:43508 - "POST /node/693b16fb15f407b7629df289 HTTP/1.0" 202 Accepted 2025-12-11 20:13:47,711 - lava_callback - INFO - Starting async job submit for node 693b16fb15f407b7629df289 2025-12-11 20:13:47,751 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:13:47,760 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b16fb15f407b7629df289 HTTP/1.1" 502 150 2025-12-11 20:13:47,760 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b16fb15f407b7629df289'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:13:47,761 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b16fb15f407b7629df289 2025-12-11 20:13:47,761 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b16fb15f407b7629df289 HTTP/1.1" 502 150 2025-12-11 20:13:47,761 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b16fb15f407b7629df289'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:13:48,321 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1406fee8383e92b742d0 2025-12-11 20:13:48,322 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1406fee8383e92b742d0 HTTP/1.1" 502 150 2025-12-11 20:13:48,322 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1406fee8383e92b742d0'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:13:49,762 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b16fb15f407b7629df289 2025-12-11 20:13:49,763 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b16fb15f407b7629df289 HTTP/1.1" 502 150 2025-12-11 20:13:49,763 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b16fb15f407b7629df289'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:13:52,323 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1406fee8383e92b742d0 2025-12-11 20:13:52,324 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1406fee8383e92b742d0 HTTP/1.1" 502 150 2025-12-11 20:13:52,324 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1406fee8383e92b742d0'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:13:53,763 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b16fb15f407b7629df289 2025-12-11 20:13:53,764 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b16fb15f407b7629df289 HTTP/1.1" 502 150 2025-12-11 20:13:53,764 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b16fb15f407b7629df289'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:55022 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:14:00,324 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1406fee8383e92b742d0 2025-12-11 20:14:00,326 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1406fee8383e92b742d0 HTTP/1.1" 502 150 2025-12-11 20:14:00,326 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1406fee8383e92b742d0'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:14:01,765 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b16fb15f407b7629df289 2025-12-11 20:14:01,766 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b16fb15f407b7629df289 HTTP/1.1" 502 150 2025-12-11 20:14:01,766 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b16fb15f407b7629df289'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:14:06,065 - lava_callback - INFO - Received callback for node 693b179915f407b7629df30b INFO: 172.21.0.1:59154 - "POST /node/693b179915f407b7629df30b HTTP/1.0" 202 Accepted 2025-12-11 20:14:06,082 - lava_callback - INFO - Starting async job submit for node 693b179915f407b7629df30b 2025-12-11 20:14:06,130 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:14:06,163 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b179915f407b7629df30b HTTP/1.1" 502 150 2025-12-11 20:14:06,163 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b179915f407b7629df30b'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:14:06,163 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b179915f407b7629df30b 2025-12-11 20:14:06,164 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b179915f407b7629df30b HTTP/1.1" 502 150 2025-12-11 20:14:06,164 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b179915f407b7629df30b'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:59168 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:14:08,164 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b179915f407b7629df30b 2025-12-11 20:14:08,165 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b179915f407b7629df30b HTTP/1.1" 502 150 2025-12-11 20:14:08,165 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b179915f407b7629df30b'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:14:08,332 - lava_callback - INFO - Received callback for node 693aee35fee8383e92b67c96 INFO: 172.21.0.1:59174 - "POST /node/693aee35fee8383e92b67c96 HTTP/1.0" 202 Accepted 2025-12-11 20:14:08,338 - lava_callback - INFO - Starting async job submit for node 693aee35fee8383e92b67c96 2025-12-11 20:14:08,359 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:14:08,371 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aee35fee8383e92b67c96 HTTP/1.1" 502 150 2025-12-11 20:14:08,371 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aee35fee8383e92b67c96'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:14:08,372 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aee35fee8383e92b67c96 2025-12-11 20:14:08,372 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aee35fee8383e92b67c96 HTTP/1.1" 502 150 2025-12-11 20:14:08,372 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aee35fee8383e92b67c96'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:14:10,373 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aee35fee8383e92b67c96 2025-12-11 20:14:10,374 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aee35fee8383e92b67c96 HTTP/1.1" 502 150 2025-12-11 20:14:10,374 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aee35fee8383e92b67c96'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:14:12,166 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b179915f407b7629df30b 2025-12-11 20:14:12,167 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b179915f407b7629df30b HTTP/1.1" 502 150 2025-12-11 20:14:12,167 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b179915f407b7629df30b'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:14:13,123 - lava_callback - INFO - Received callback for node 693b0acefee8383e92b71b2b INFO: 172.21.0.1:33370 - "POST /node/693b0acefee8383e92b71b2b HTTP/1.0" 202 Accepted 2025-12-11 20:14:13,131 - lava_callback - INFO - Starting async job submit for node 693b0acefee8383e92b71b2b 2025-12-11 20:14:13,152 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:14:13,162 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0acefee8383e92b71b2b HTTP/1.1" 502 150 2025-12-11 20:14:13,162 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0acefee8383e92b71b2b'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:14:13,162 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0acefee8383e92b71b2b 2025-12-11 20:14:13,163 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0acefee8383e92b71b2b HTTP/1.1" 502 150 2025-12-11 20:14:13,163 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0acefee8383e92b71b2b'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:14:14,374 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aee35fee8383e92b67c96 2025-12-11 20:14:14,376 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aee35fee8383e92b67c96 HTTP/1.1" 502 150 2025-12-11 20:14:14,376 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aee35fee8383e92b67c96'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:33376 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 20:14:15,163 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0acefee8383e92b71b2b 2025-12-11 20:14:15,164 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0acefee8383e92b71b2b HTTP/1.1" 502 150 2025-12-11 20:14:15,164 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0acefee8383e92b71b2b'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:14:16,326 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1406fee8383e92b742d0 2025-12-11 20:14:16,327 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1406fee8383e92b742d0 HTTP/1.1" 502 150 2025-12-11 20:14:16,327 - lava_callback - ERROR - Error processing callback for node 693b1406fee8383e92b742d0: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1406fee8383e92b742d0 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1406fee8383e92b742d0 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1406fee8383e92b742d0 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:33390 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:14:17,766 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b16fb15f407b7629df289 2025-12-11 20:14:17,767 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b16fb15f407b7629df289 HTTP/1.1" 502 150 2025-12-11 20:14:17,767 - lava_callback - ERROR - Error processing callback for node 693b16fb15f407b7629df289: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b16fb15f407b7629df289 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b16fb15f407b7629df289 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b16fb15f407b7629df289 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:14:19,165 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0acefee8383e92b71b2b 2025-12-11 20:14:19,170 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0acefee8383e92b71b2b HTTP/1.1" 502 150 2025-12-11 20:14:19,170 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0acefee8383e92b71b2b'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:14:20,167 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b179915f407b7629df30b 2025-12-11 20:14:20,169 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b179915f407b7629df30b HTTP/1.1" 502 150 2025-12-11 20:14:20,169 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b179915f407b7629df30b'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:14:22,376 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aee35fee8383e92b67c96 2025-12-11 20:14:22,377 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aee35fee8383e92b67c96 HTTP/1.1" 502 150 2025-12-11 20:14:22,377 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aee35fee8383e92b67c96'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:59596 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:14:27,170 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0acefee8383e92b71b2b 2025-12-11 20:14:27,172 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0acefee8383e92b71b2b HTTP/1.1" 502 150 2025-12-11 20:14:27,172 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0acefee8383e92b71b2b'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:59604 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:14:33,642 - lava_callback - INFO - Received callback for node 693b1bc315f407b7629e02c6 INFO: 172.21.0.1:39008 - "POST /node/693b1bc315f407b7629e02c6 HTTP/1.0" 202 Accepted 2025-12-11 20:14:33,653 - lava_callback - INFO - Starting async job submit for node 693b1bc315f407b7629e02c6 2025-12-11 20:14:33,692 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:14:33,705 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bc315f407b7629e02c6 HTTP/1.1" 502 150 2025-12-11 20:14:33,705 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bc315f407b7629e02c6'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:14:33,705 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bc315f407b7629e02c6 2025-12-11 20:14:33,706 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bc315f407b7629e02c6 HTTP/1.1" 502 150 2025-12-11 20:14:33,706 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bc315f407b7629e02c6'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:14:35,706 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bc315f407b7629e02c6 2025-12-11 20:14:35,707 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bc315f407b7629e02c6 HTTP/1.1" 502 150 2025-12-11 20:14:35,707 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bc315f407b7629e02c6'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:14:36,169 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b179915f407b7629df30b 2025-12-11 20:14:36,171 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b179915f407b7629df30b HTTP/1.1" 502 150 2025-12-11 20:14:36,171 - lava_callback - ERROR - Error processing callback for node 693b179915f407b7629df30b: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b179915f407b7629df30b (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b179915f407b7629df30b (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b179915f407b7629df30b (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:39022 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:14:38,377 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aee35fee8383e92b67c96 2025-12-11 20:14:38,379 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aee35fee8383e92b67c96 HTTP/1.1" 502 150 2025-12-11 20:14:38,379 - lava_callback - ERROR - Error processing callback for node 693aee35fee8383e92b67c96: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aee35fee8383e92b67c96 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aee35fee8383e92b67c96 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aee35fee8383e92b67c96 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:14:39,707 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bc315f407b7629e02c6 2025-12-11 20:14:39,709 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bc315f407b7629e02c6 HTTP/1.1" 502 150 2025-12-11 20:14:39,709 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bc315f407b7629e02c6'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:14:43,172 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0acefee8383e92b71b2b 2025-12-11 20:14:43,173 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0acefee8383e92b71b2b HTTP/1.1" 502 150 2025-12-11 20:14:43,173 - lava_callback - ERROR - Error processing callback for node 693b0acefee8383e92b71b2b: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0acefee8383e92b71b2b (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0acefee8383e92b71b2b (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0acefee8383e92b71b2b (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:55038 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:14:47,709 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bc315f407b7629e02c6 2025-12-11 20:14:47,710 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bc315f407b7629e02c6 HTTP/1.1" 502 150 2025-12-11 20:14:47,710 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bc315f407b7629e02c6'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:41948 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:15:03,711 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bc315f407b7629e02c6 2025-12-11 20:15:03,712 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bc315f407b7629e02c6 HTTP/1.1" 502 150 2025-12-11 20:15:03,712 - lava_callback - ERROR - Error processing callback for node 693b1bc315f407b7629e02c6: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1bc315f407b7629e02c6 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1bc315f407b7629e02c6 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1bc315f407b7629e02c6 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:53594 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49426 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:49430 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58964 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58966 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50696 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:15:46,268 - lava_callback - INFO - Received callback for node 693b179915f407b7629df30f INFO: 172.21.0.1:34616 - "POST /node/693b179915f407b7629df30f HTTP/1.0" 202 Accepted 2025-12-11 20:15:46,278 - lava_callback - INFO - Starting async job submit for node 693b179915f407b7629df30f 2025-12-11 20:15:46,318 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:15:46,350 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b179915f407b7629df30f HTTP/1.1" 502 150 2025-12-11 20:15:46,350 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b179915f407b7629df30f'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:15:46,350 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b179915f407b7629df30f 2025-12-11 20:15:46,351 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b179915f407b7629df30f HTTP/1.1" 502 150 2025-12-11 20:15:46,351 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b179915f407b7629df30f'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:34624 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:15:47,466 - lava_callback - INFO - Received callback for node 693aef09fee8383e92b687a9 INFO: 172.21.0.1:34632 - "POST /node/693aef09fee8383e92b687a9 HTTP/1.0" 202 Accepted 2025-12-11 20:15:47,472 - lava_callback - INFO - Starting async job submit for node 693aef09fee8383e92b687a9 2025-12-11 20:15:47,493 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:15:47,525 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aef09fee8383e92b687a9 HTTP/1.1" 502 150 2025-12-11 20:15:47,525 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aef09fee8383e92b687a9'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:15:47,525 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aef09fee8383e92b687a9 2025-12-11 20:15:47,526 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aef09fee8383e92b687a9 HTTP/1.1" 502 150 2025-12-11 20:15:47,526 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aef09fee8383e92b687a9'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:15:48,352 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b179915f407b7629df30f 2025-12-11 20:15:48,359 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b179915f407b7629df30f HTTP/1.1" 502 150 2025-12-11 20:15:48,359 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b179915f407b7629df30f'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:15:49,526 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aef09fee8383e92b687a9 2025-12-11 20:15:49,528 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aef09fee8383e92b687a9 HTTP/1.1" 502 150 2025-12-11 20:15:49,528 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aef09fee8383e92b687a9'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:15:52,360 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b179915f407b7629df30f 2025-12-11 20:15:52,361 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b179915f407b7629df30f HTTP/1.1" 502 150 2025-12-11 20:15:52,361 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b179915f407b7629df30f'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:15:53,528 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aef09fee8383e92b687a9 2025-12-11 20:15:53,530 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aef09fee8383e92b687a9 HTTP/1.1" 502 150 2025-12-11 20:15:53,530 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aef09fee8383e92b687a9'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:37920 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:16:00,361 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b179915f407b7629df30f 2025-12-11 20:16:00,362 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b179915f407b7629df30f HTTP/1.1" 502 150 2025-12-11 20:16:00,362 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b179915f407b7629df30f'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:16:01,530 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aef09fee8383e92b687a9 2025-12-11 20:16:01,531 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aef09fee8383e92b687a9 HTTP/1.1" 502 150 2025-12-11 20:16:01,531 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aef09fee8383e92b687a9'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:16:05,046 - lava_callback - INFO - Received callback for node 693b081bfee8383e92b711b1 INFO: 172.21.0.1:43434 - "POST /node/693b081bfee8383e92b711b1 HTTP/1.0" 202 Accepted 2025-12-11 20:16:05,055 - lava_callback - INFO - Starting async job submit for node 693b081bfee8383e92b711b1 2025-12-11 20:16:05,114 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:16:05,124 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b081bfee8383e92b711b1 HTTP/1.1" 502 150 2025-12-11 20:16:05,124 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b081bfee8383e92b711b1'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:16:05,124 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b081bfee8383e92b711b1 2025-12-11 20:16:05,125 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b081bfee8383e92b711b1 HTTP/1.1" 502 150 2025-12-11 20:16:05,125 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b081bfee8383e92b711b1'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:16:06,076 - lava_callback - INFO - Received callback for node 693af5f8fee8383e92b6b67a INFO: 172.21.0.1:43448 - "POST /node/693af5f8fee8383e92b6b67a HTTP/1.0" 202 Accepted 2025-12-11 20:16:06,082 - lava_callback - INFO - Starting async job submit for node 693af5f8fee8383e92b6b67a 2025-12-11 20:16:06,100 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:16:06,129 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af5f8fee8383e92b6b67a HTTP/1.1" 502 150 2025-12-11 20:16:06,129 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af5f8fee8383e92b6b67a'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:16:06,129 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af5f8fee8383e92b6b67a 2025-12-11 20:16:06,129 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af5f8fee8383e92b6b67a HTTP/1.1" 502 150 2025-12-11 20:16:06,130 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af5f8fee8383e92b6b67a'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:43450 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:16:06,942 - lava_callback - INFO - Received callback for node 693ae872fee8383e92b65944 INFO: 172.21.0.1:43460 - "POST /node/693ae872fee8383e92b65944 HTTP/1.0" 202 Accepted 2025-12-11 20:16:06,949 - lava_callback - INFO - Starting async job submit for node 693ae872fee8383e92b65944 2025-12-11 20:16:06,973 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:16:06,983 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae872fee8383e92b65944 HTTP/1.1" 502 150 2025-12-11 20:16:06,983 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae872fee8383e92b65944'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:16:06,983 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae872fee8383e92b65944 2025-12-11 20:16:06,984 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae872fee8383e92b65944 HTTP/1.1" 502 150 2025-12-11 20:16:06,984 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae872fee8383e92b65944'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:16:07,125 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b081bfee8383e92b711b1 2025-12-11 20:16:07,126 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b081bfee8383e92b711b1 HTTP/1.1" 502 150 2025-12-11 20:16:07,126 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b081bfee8383e92b711b1'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:16:08,130 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af5f8fee8383e92b6b67a 2025-12-11 20:16:08,131 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af5f8fee8383e92b6b67a HTTP/1.1" 502 150 2025-12-11 20:16:08,131 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af5f8fee8383e92b6b67a'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:16:08,984 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae872fee8383e92b65944 2025-12-11 20:16:08,985 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae872fee8383e92b65944 HTTP/1.1" 502 150 2025-12-11 20:16:08,986 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae872fee8383e92b65944'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:16:11,126 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b081bfee8383e92b711b1 2025-12-11 20:16:11,128 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b081bfee8383e92b711b1 HTTP/1.1" 502 150 2025-12-11 20:16:11,128 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b081bfee8383e92b711b1'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:16:12,131 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af5f8fee8383e92b6b67a 2025-12-11 20:16:12,132 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af5f8fee8383e92b6b67a HTTP/1.1" 502 150 2025-12-11 20:16:12,133 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af5f8fee8383e92b6b67a'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:16:12,986 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae872fee8383e92b65944 2025-12-11 20:16:12,987 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae872fee8383e92b65944 HTTP/1.1" 502 150 2025-12-11 20:16:12,987 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae872fee8383e92b65944'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:41398 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 20:16:16,362 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b179915f407b7629df30f 2025-12-11 20:16:16,364 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b179915f407b7629df30f HTTP/1.1" 502 150 2025-12-11 20:16:16,364 - lava_callback - ERROR - Error processing callback for node 693b179915f407b7629df30f: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b179915f407b7629df30f (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b179915f407b7629df30f (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b179915f407b7629df30f (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:41414 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:16:17,531 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aef09fee8383e92b687a9 2025-12-11 20:16:17,533 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aef09fee8383e92b687a9 HTTP/1.1" 502 150 2025-12-11 20:16:17,533 - lava_callback - ERROR - Error processing callback for node 693aef09fee8383e92b687a9: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aef09fee8383e92b687a9 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aef09fee8383e92b687a9 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aef09fee8383e92b687a9 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:16:19,128 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b081bfee8383e92b711b1 2025-12-11 20:16:19,129 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b081bfee8383e92b711b1 HTTP/1.1" 502 150 2025-12-11 20:16:19,129 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b081bfee8383e92b711b1'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:16:20,133 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af5f8fee8383e92b6b67a 2025-12-11 20:16:20,134 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af5f8fee8383e92b6b67a HTTP/1.1" 502 150 2025-12-11 20:16:20,134 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af5f8fee8383e92b6b67a'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:16:20,987 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae872fee8383e92b65944 2025-12-11 20:16:20,989 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae872fee8383e92b65944 HTTP/1.1" 502 150 2025-12-11 20:16:20,989 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae872fee8383e92b65944'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:60358 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60366 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:16:31,307 - lava_callback - INFO - Received callback for node 693b01affee8383e92b6ee07 INFO: 172.21.0.1:58778 - "POST /node/693b01affee8383e92b6ee07 HTTP/1.0" 202 Accepted 2025-12-11 20:16:31,318 - lava_callback - INFO - Starting async job submit for node 693b01affee8383e92b6ee07 2025-12-11 20:16:31,382 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:16:31,397 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b01affee8383e92b6ee07 HTTP/1.1" 502 150 2025-12-11 20:16:31,397 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b01affee8383e92b6ee07'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:16:31,397 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b01affee8383e92b6ee07 2025-12-11 20:16:31,398 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b01affee8383e92b6ee07 HTTP/1.1" 502 150 2025-12-11 20:16:31,398 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b01affee8383e92b6ee07'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:16:33,398 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b01affee8383e92b6ee07 2025-12-11 20:16:33,400 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b01affee8383e92b6ee07 HTTP/1.1" 502 150 2025-12-11 20:16:33,400 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b01affee8383e92b6ee07'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:16:35,130 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b081bfee8383e92b711b1 2025-12-11 20:16:35,131 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b081bfee8383e92b711b1 HTTP/1.1" 502 150 2025-12-11 20:16:35,131 - lava_callback - ERROR - Error processing callback for node 693b081bfee8383e92b711b1: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b081bfee8383e92b711b1 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b081bfee8383e92b711b1 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b081bfee8383e92b711b1 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:16:36,134 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af5f8fee8383e92b6b67a 2025-12-11 20:16:36,135 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af5f8fee8383e92b6b67a HTTP/1.1" 502 150 2025-12-11 20:16:36,136 - lava_callback - ERROR - Error processing callback for node 693af5f8fee8383e92b6b67a: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af5f8fee8383e92b6b67a (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af5f8fee8383e92b6b67a (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af5f8fee8383e92b6b67a (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:58788 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:16:36,989 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae872fee8383e92b65944 2025-12-11 20:16:36,990 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae872fee8383e92b65944 HTTP/1.1" 502 150 2025-12-11 20:16:36,990 - lava_callback - ERROR - Error processing callback for node 693ae872fee8383e92b65944: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae872fee8383e92b65944 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae872fee8383e92b65944 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae872fee8383e92b65944 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:16:37,400 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b01affee8383e92b6ee07 2025-12-11 20:16:37,401 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b01affee8383e92b6ee07 HTTP/1.1" 502 150 2025-12-11 20:16:37,401 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b01affee8383e92b6ee07'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:16:38,102 - lava_callback - INFO - Received callback for node 693b1bcf15f407b7629e02dc INFO: 172.21.0.1:58794 - "POST /node/693b1bcf15f407b7629e02dc HTTP/1.0" 202 Accepted 2025-12-11 20:16:38,109 - lava_callback - INFO - Starting async job submit for node 693b1bcf15f407b7629e02dc 2025-12-11 20:16:38,133 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:16:38,143 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bcf15f407b7629e02dc HTTP/1.1" 502 150 2025-12-11 20:16:38,144 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bcf15f407b7629e02dc'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:16:38,144 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bcf15f407b7629e02dc 2025-12-11 20:16:38,144 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bcf15f407b7629e02dc HTTP/1.1" 502 150 2025-12-11 20:16:38,145 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bcf15f407b7629e02dc'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:16:40,145 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bcf15f407b7629e02dc 2025-12-11 20:16:40,146 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bcf15f407b7629e02dc HTTP/1.1" 502 150 2025-12-11 20:16:40,146 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bcf15f407b7629e02dc'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:16:44,146 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bcf15f407b7629e02dc 2025-12-11 20:16:44,148 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bcf15f407b7629e02dc HTTP/1.1" 502 150 2025-12-11 20:16:44,148 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bcf15f407b7629e02dc'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:16:45,401 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b01affee8383e92b6ee07 2025-12-11 20:16:45,402 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b01affee8383e92b6ee07 HTTP/1.1" 502 150 2025-12-11 20:16:45,403 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b01affee8383e92b6ee07'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:55014 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:16:47,053 - lava_callback - INFO - Received callback for node 6938868e63d980a97480488a INFO: 172.21.0.1:55020 - "POST /node/6938868e63d980a97480488a HTTP/1.0" 202 Accepted 2025-12-11 20:16:47,063 - lava_callback - INFO - Starting async job submit for node 6938868e63d980a97480488a 2025-12-11 20:16:47,103 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:16:47,120 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938868e63d980a97480488a HTTP/1.1" 502 150 2025-12-11 20:16:47,120 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938868e63d980a97480488a'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:16:47,121 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938868e63d980a97480488a 2025-12-11 20:16:47,122 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938868e63d980a97480488a HTTP/1.1" 502 150 2025-12-11 20:16:47,122 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938868e63d980a97480488a'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:16:49,122 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938868e63d980a97480488a 2025-12-11 20:16:49,124 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938868e63d980a97480488a HTTP/1.1" 502 150 2025-12-11 20:16:49,124 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938868e63d980a97480488a'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:16:52,148 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bcf15f407b7629e02dc 2025-12-11 20:16:52,154 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bcf15f407b7629e02dc HTTP/1.1" 502 150 2025-12-11 20:16:52,154 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bcf15f407b7629e02dc'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:16:53,124 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938868e63d980a97480488a 2025-12-11 20:16:53,125 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938868e63d980a97480488a HTTP/1.1" 502 150 2025-12-11 20:16:53,125 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938868e63d980a97480488a'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:40022 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:17:01,126 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938868e63d980a97480488a 2025-12-11 20:17:01,127 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938868e63d980a97480488a HTTP/1.1" 502 150 2025-12-11 20:17:01,127 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938868e63d980a97480488a'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:17:01,403 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b01affee8383e92b6ee07 2025-12-11 20:17:01,404 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b01affee8383e92b6ee07 HTTP/1.1" 502 150 2025-12-11 20:17:01,404 - lava_callback - ERROR - Error processing callback for node 693b01affee8383e92b6ee07: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b01affee8383e92b6ee07 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b01affee8383e92b6ee07 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b01affee8383e92b6ee07 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:34748 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:17:08,154 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bcf15f407b7629e02dc 2025-12-11 20:17:08,155 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bcf15f407b7629e02dc HTTP/1.1" 502 150 2025-12-11 20:17:08,156 - lava_callback - ERROR - Error processing callback for node 693b1bcf15f407b7629e02dc: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1bcf15f407b7629e02dc (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1bcf15f407b7629e02dc (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1bcf15f407b7629e02dc (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:17:12,327 - lava_callback - INFO - Received callback for node 693b0b16fee8383e92b71c26 INFO: 172.21.0.1:50530 - "POST /node/693b0b16fee8383e92b71c26 HTTP/1.0" 202 Accepted 2025-12-11 20:17:12,336 - lava_callback - INFO - Starting async job submit for node 693b0b16fee8383e92b71c26 2025-12-11 20:17:12,356 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:17:12,368 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0b16fee8383e92b71c26 HTTP/1.1" 502 150 2025-12-11 20:17:12,368 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0b16fee8383e92b71c26'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:17:12,368 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0b16fee8383e92b71c26 2025-12-11 20:17:12,368 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0b16fee8383e92b71c26 HTTP/1.1" 502 150 2025-12-11 20:17:12,368 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0b16fee8383e92b71c26'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:17:14,369 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0b16fee8383e92b71c26 2025-12-11 20:17:14,370 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0b16fee8383e92b71c26 HTTP/1.1" 502 150 2025-12-11 20:17:14,370 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0b16fee8383e92b71c26'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:50538 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:50554 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:17:17,127 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938868e63d980a97480488a 2025-12-11 20:17:17,130 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938868e63d980a97480488a HTTP/1.1" 502 150 2025-12-11 20:17:17,130 - lava_callback - ERROR - Error processing callback for node 6938868e63d980a97480488a: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938868e63d980a97480488a (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938868e63d980a97480488a (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938868e63d980a97480488a (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:17:18,370 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0b16fee8383e92b71c26 2025-12-11 20:17:18,372 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0b16fee8383e92b71c26 HTTP/1.1" 502 150 2025-12-11 20:17:18,372 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0b16fee8383e92b71c26'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:17:26,372 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0b16fee8383e92b71c26 2025-12-11 20:17:26,373 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0b16fee8383e92b71c26 HTTP/1.1" 502 150 2025-12-11 20:17:26,374 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0b16fee8383e92b71c26'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:34830 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34840 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:17:30,492 - lava_callback - INFO - Received callback for node 693aef6cfee8383e92b68a17 INFO: 172.21.0.1:48532 - "POST /node/693aef6cfee8383e92b68a17 HTTP/1.0" 202 Accepted 2025-12-11 20:17:30,503 - lava_callback - INFO - Starting async job submit for node 693aef6cfee8383e92b68a17 2025-12-11 20:17:30,545 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:17:30,577 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aef6cfee8383e92b68a17 HTTP/1.1" 502 150 2025-12-11 20:17:30,577 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aef6cfee8383e92b68a17'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:17:30,577 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aef6cfee8383e92b68a17 2025-12-11 20:17:30,578 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aef6cfee8383e92b68a17 HTTP/1.1" 502 150 2025-12-11 20:17:30,578 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aef6cfee8383e92b68a17'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:17:32,579 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aef6cfee8383e92b68a17 2025-12-11 20:17:32,580 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aef6cfee8383e92b68a17 HTTP/1.1" 502 150 2025-12-11 20:17:32,580 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aef6cfee8383e92b68a17'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:48536 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:17:36,580 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aef6cfee8383e92b68a17 2025-12-11 20:17:36,582 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aef6cfee8383e92b68a17 HTTP/1.1" 502 150 2025-12-11 20:17:36,582 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aef6cfee8383e92b68a17'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:17:42,374 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0b16fee8383e92b71c26 2025-12-11 20:17:42,375 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0b16fee8383e92b71c26 HTTP/1.1" 502 150 2025-12-11 20:17:42,375 - lava_callback - ERROR - Error processing callback for node 693b0b16fee8383e92b71c26: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0b16fee8383e92b71c26 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0b16fee8383e92b71c26 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0b16fee8383e92b71c26 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:17:44,582 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aef6cfee8383e92b68a17 2025-12-11 20:17:44,583 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aef6cfee8383e92b68a17 HTTP/1.1" 502 150 2025-12-11 20:17:44,583 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aef6cfee8383e92b68a17'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:49178 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56122 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:18:00,584 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aef6cfee8383e92b68a17 2025-12-11 20:18:00,585 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aef6cfee8383e92b68a17 HTTP/1.1" 502 150 2025-12-11 20:18:00,585 - lava_callback - ERROR - Error processing callback for node 693aef6cfee8383e92b68a17: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aef6cfee8383e92b68a17 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aef6cfee8383e92b68a17 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aef6cfee8383e92b68a17 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:18:06,267 - lava_callback - INFO - Received callback for node 693af725fee8383e92b6bf10 INFO: 172.21.0.1:39944 - "POST /node/693af725fee8383e92b6bf10 HTTP/1.0" 202 Accepted 2025-12-11 20:18:06,278 - lava_callback - INFO - Starting async job submit for node 693af725fee8383e92b6bf10 2025-12-11 20:18:06,351 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:18:06,365 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af725fee8383e92b6bf10 HTTP/1.1" 502 150 2025-12-11 20:18:06,365 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af725fee8383e92b6bf10'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:18:06,365 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af725fee8383e92b6bf10 2025-12-11 20:18:06,371 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af725fee8383e92b6bf10 HTTP/1.1" 502 150 2025-12-11 20:18:06,371 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af725fee8383e92b6bf10'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:39958 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:18:08,371 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af725fee8383e92b6bf10 2025-12-11 20:18:08,373 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af725fee8383e92b6bf10 HTTP/1.1" 502 150 2025-12-11 20:18:08,373 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af725fee8383e92b6bf10'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:18:12,373 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af725fee8383e92b6bf10 2025-12-11 20:18:12,374 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af725fee8383e92b6bf10 HTTP/1.1" 502 150 2025-12-11 20:18:12,374 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af725fee8383e92b6bf10'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:35572 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:35574 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:18:20,374 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af725fee8383e92b6bf10 2025-12-11 20:18:20,376 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af725fee8383e92b6bf10 HTTP/1.1" 502 150 2025-12-11 20:18:20,376 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af725fee8383e92b6bf10'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:18:26,405 - lava_callback - INFO - Received callback for node 693b0e14fee8383e92b7248d INFO: 172.21.0.1:58946 - "POST /node/693b0e14fee8383e92b7248d HTTP/1.0" 202 Accepted 2025-12-11 20:18:26,416 - lava_callback - INFO - Starting async job submit for node 693b0e14fee8383e92b7248d 2025-12-11 20:18:26,481 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:18:26,507 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e14fee8383e92b7248d HTTP/1.1" 502 150 2025-12-11 20:18:26,507 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e14fee8383e92b7248d'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:18:26,508 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e14fee8383e92b7248d 2025-12-11 20:18:26,508 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e14fee8383e92b7248d HTTP/1.1" 502 150 2025-12-11 20:18:26,508 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e14fee8383e92b7248d'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:58948 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58956 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:18:28,509 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e14fee8383e92b7248d 2025-12-11 20:18:28,512 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e14fee8383e92b7248d HTTP/1.1" 502 150 2025-12-11 20:18:28,512 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e14fee8383e92b7248d'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:18:29,638 - lava_callback - INFO - Received callback for node 693b1bd515f407b7629e02ed INFO: 172.21.0.1:58972 - "POST /node/693b1bd515f407b7629e02ed HTTP/1.0" 202 Accepted 2025-12-11 20:18:29,645 - lava_callback - INFO - Starting async job submit for node 693b1bd515f407b7629e02ed 2025-12-11 20:18:29,676 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:18:29,699 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bd515f407b7629e02ed HTTP/1.1" 502 150 2025-12-11 20:18:29,700 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bd515f407b7629e02ed'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:18:29,700 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bd515f407b7629e02ed 2025-12-11 20:18:29,700 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bd515f407b7629e02ed HTTP/1.1" 502 150 2025-12-11 20:18:29,700 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bd515f407b7629e02ed'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:18:31,701 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bd515f407b7629e02ed 2025-12-11 20:18:31,702 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bd515f407b7629e02ed HTTP/1.1" 502 150 2025-12-11 20:18:31,702 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bd515f407b7629e02ed'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:18:32,512 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e14fee8383e92b7248d 2025-12-11 20:18:32,513 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e14fee8383e92b7248d HTTP/1.1" 502 150 2025-12-11 20:18:32,513 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e14fee8383e92b7248d'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:18:35,702 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bd515f407b7629e02ed 2025-12-11 20:18:35,704 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bd515f407b7629e02ed HTTP/1.1" 502 150 2025-12-11 20:18:35,704 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bd515f407b7629e02ed'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:18:36,376 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af725fee8383e92b6bf10 2025-12-11 20:18:36,377 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af725fee8383e92b6bf10 HTTP/1.1" 502 150 2025-12-11 20:18:36,377 - lava_callback - ERROR - Error processing callback for node 693af725fee8383e92b6bf10: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af725fee8383e92b6bf10 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af725fee8383e92b6bf10 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af725fee8383e92b6bf10 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:39840 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:18:40,514 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e14fee8383e92b7248d 2025-12-11 20:18:40,515 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e14fee8383e92b7248d HTTP/1.1" 502 150 2025-12-11 20:18:40,515 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e14fee8383e92b7248d'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:18:42,491 - lava_callback - INFO - Received callback for node 6938883163d980a9748048aa INFO: 172.21.0.1:53372 - "POST /node/6938883163d980a9748048aa HTTP/1.0" 202 Accepted 2025-12-11 20:18:42,497 - lava_callback - INFO - Starting async job submit for node 6938883163d980a9748048aa 2025-12-11 20:18:42,529 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:18:42,559 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938883163d980a9748048aa HTTP/1.1" 502 150 2025-12-11 20:18:42,559 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938883163d980a9748048aa'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:18:42,559 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938883163d980a9748048aa 2025-12-11 20:18:42,560 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938883163d980a9748048aa HTTP/1.1" 502 150 2025-12-11 20:18:42,560 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938883163d980a9748048aa'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:18:43,704 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bd515f407b7629e02ed 2025-12-11 20:18:43,706 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bd515f407b7629e02ed HTTP/1.1" 502 150 2025-12-11 20:18:43,706 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bd515f407b7629e02ed'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:18:44,560 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938883163d980a9748048aa 2025-12-11 20:18:44,561 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938883163d980a9748048aa HTTP/1.1" 502 150 2025-12-11 20:18:44,561 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938883163d980a9748048aa'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:53376 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:18:48,562 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938883163d980a9748048aa 2025-12-11 20:18:48,563 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938883163d980a9748048aa HTTP/1.1" 502 150 2025-12-11 20:18:48,563 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938883163d980a9748048aa'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:18:56,515 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e14fee8383e92b7248d 2025-12-11 20:18:56,516 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e14fee8383e92b7248d HTTP/1.1" 502 150 2025-12-11 20:18:56,516 - lava_callback - ERROR - Error processing callback for node 693b0e14fee8383e92b7248d: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e14fee8383e92b7248d (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e14fee8383e92b7248d (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e14fee8383e92b7248d (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:50994 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:18:56,563 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938883163d980a9748048aa 2025-12-11 20:18:56,564 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938883163d980a9748048aa HTTP/1.1" 502 150 2025-12-11 20:18:56,564 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938883163d980a9748048aa'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:18:59,706 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bd515f407b7629e02ed 2025-12-11 20:18:59,707 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bd515f407b7629e02ed HTTP/1.1" 502 150 2025-12-11 20:18:59,708 - lava_callback - ERROR - Error processing callback for node 693b1bd515f407b7629e02ed: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1bd515f407b7629e02ed (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1bd515f407b7629e02ed (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1bd515f407b7629e02ed (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:19:05,590 - lava_callback - INFO - Received callback for node 693aefb9fee8383e92b68c78 INFO: 172.21.0.1:48684 - "POST /node/693aefb9fee8383e92b68c78 HTTP/1.0" 202 Accepted 2025-12-11 20:19:05,600 - lava_callback - INFO - Starting async job submit for node 693aefb9fee8383e92b68c78 2025-12-11 20:19:05,644 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:19:05,660 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aefb9fee8383e92b68c78 HTTP/1.1" 502 150 2025-12-11 20:19:05,660 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aefb9fee8383e92b68c78'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:19:05,660 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aefb9fee8383e92b68c78 2025-12-11 20:19:05,661 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aefb9fee8383e92b68c78 HTTP/1.1" 502 150 2025-12-11 20:19:05,661 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aefb9fee8383e92b68c78'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:48694 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:19:07,662 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aefb9fee8383e92b68c78 2025-12-11 20:19:07,663 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aefb9fee8383e92b68c78 HTTP/1.1" 502 150 2025-12-11 20:19:07,663 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aefb9fee8383e92b68c78'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:19:08,757 - lava_callback - INFO - Received callback for node 693894e863d980a9748051b3 INFO: 172.21.0.1:48702 - "POST /node/693894e863d980a9748051b3 HTTP/1.0" 202 Accepted 2025-12-11 20:19:08,767 - lava_callback - INFO - Starting async job submit for node 693894e863d980a9748051b3 2025-12-11 20:19:08,824 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:19:08,835 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693894e863d980a9748051b3 HTTP/1.1" 502 150 2025-12-11 20:19:08,835 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693894e863d980a9748051b3'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:19:08,835 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693894e863d980a9748051b3 2025-12-11 20:19:08,836 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693894e863d980a9748051b3 HTTP/1.1" 502 150 2025-12-11 20:19:08,836 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693894e863d980a9748051b3'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:19:10,836 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693894e863d980a9748051b3 2025-12-11 20:19:10,838 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693894e863d980a9748051b3 HTTP/1.1" 502 150 2025-12-11 20:19:10,838 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693894e863d980a9748051b3'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:19:11,663 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aefb9fee8383e92b68c78 2025-12-11 20:19:11,664 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aefb9fee8383e92b68c78 HTTP/1.1" 502 150 2025-12-11 20:19:11,664 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aefb9fee8383e92b68c78'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:19:12,564 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938883163d980a9748048aa 2025-12-11 20:19:12,566 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938883163d980a9748048aa HTTP/1.1" 502 150 2025-12-11 20:19:12,566 - lava_callback - ERROR - Error processing callback for node 6938883163d980a9748048aa: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938883163d980a9748048aa (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938883163d980a9748048aa (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938883163d980a9748048aa (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:19:14,838 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693894e863d980a9748051b3 2025-12-11 20:19:14,839 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693894e863d980a9748051b3 HTTP/1.1" 502 150 2025-12-11 20:19:14,840 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693894e863d980a9748051b3'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:52468 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:52472 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:19:19,665 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aefb9fee8383e92b68c78 2025-12-11 20:19:19,666 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aefb9fee8383e92b68c78 HTTP/1.1" 502 150 2025-12-11 20:19:19,666 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aefb9fee8383e92b68c78'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:19:22,840 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693894e863d980a9748051b3 2025-12-11 20:19:22,841 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693894e863d980a9748051b3 HTTP/1.1" 502 150 2025-12-11 20:19:22,841 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693894e863d980a9748051b3'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:19:26,123 - lava_callback - INFO - Received callback for node 693b179915f407b7629df30d INFO: 172.21.0.1:53374 - "POST /node/693b179915f407b7629df30d HTTP/1.0" 202 Accepted 2025-12-11 20:19:26,138 - lava_callback - INFO - Starting async job submit for node 693b179915f407b7629df30d 2025-12-11 20:19:26,192 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:19:26,215 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b179915f407b7629df30d HTTP/1.1" 502 150 2025-12-11 20:19:26,215 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b179915f407b7629df30d'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:19:26,215 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b179915f407b7629df30d 2025-12-11 20:19:26,216 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b179915f407b7629df30d HTTP/1.1" 502 150 2025-12-11 20:19:26,216 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b179915f407b7629df30d'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:53382 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:19:27,187 - lava_callback - INFO - Received callback for node 693ae86dfee8383e92b65929 INFO: 172.21.0.1:53388 - "POST /node/693ae86dfee8383e92b65929 HTTP/1.0" 202 Accepted 2025-12-11 20:19:27,211 - lava_callback - INFO - Starting async job submit for node 693ae86dfee8383e92b65929 2025-12-11 20:19:27,455 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 INFO: 172.21.0.1:53404 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:19:27,468 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae86dfee8383e92b65929 HTTP/1.1" 502 150 2025-12-11 20:19:27,469 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae86dfee8383e92b65929'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:19:27,469 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae86dfee8383e92b65929 2025-12-11 20:19:27,470 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae86dfee8383e92b65929 HTTP/1.1" 502 150 2025-12-11 20:19:27,470 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae86dfee8383e92b65929'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:19:28,216 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b179915f407b7629df30d 2025-12-11 20:19:28,217 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b179915f407b7629df30d HTTP/1.1" 502 150 2025-12-11 20:19:28,218 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b179915f407b7629df30d'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:19:29,471 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae86dfee8383e92b65929 2025-12-11 20:19:29,472 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae86dfee8383e92b65929 HTTP/1.1" 502 150 2025-12-11 20:19:29,472 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae86dfee8383e92b65929'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:19:32,218 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b179915f407b7629df30d 2025-12-11 20:19:32,219 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b179915f407b7629df30d HTTP/1.1" 502 150 2025-12-11 20:19:32,219 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b179915f407b7629df30d'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:19:33,472 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae86dfee8383e92b65929 2025-12-11 20:19:33,474 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae86dfee8383e92b65929 HTTP/1.1" 502 150 2025-12-11 20:19:33,474 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae86dfee8383e92b65929'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:19:35,666 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aefb9fee8383e92b68c78 2025-12-11 20:19:35,668 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aefb9fee8383e92b68c78 HTTP/1.1" 502 150 2025-12-11 20:19:35,668 - lava_callback - ERROR - Error processing callback for node 693aefb9fee8383e92b68c78: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aefb9fee8383e92b68c78 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aefb9fee8383e92b68c78 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aefb9fee8383e92b68c78 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:43970 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:19:38,841 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693894e863d980a9748051b3 2025-12-11 20:19:38,846 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693894e863d980a9748051b3 HTTP/1.1" 502 150 2025-12-11 20:19:38,846 - lava_callback - ERROR - Error processing callback for node 693894e863d980a9748051b3: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693894e863d980a9748051b3 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693894e863d980a9748051b3 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693894e863d980a9748051b3 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:19:40,219 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b179915f407b7629df30d 2025-12-11 20:19:40,221 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b179915f407b7629df30d HTTP/1.1" 502 150 2025-12-11 20:19:40,221 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b179915f407b7629df30d'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:19:41,474 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae86dfee8383e92b65929 2025-12-11 20:19:41,475 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae86dfee8383e92b65929 HTTP/1.1" 502 150 2025-12-11 20:19:41,475 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae86dfee8383e92b65929'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:38036 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:19:56,221 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b179915f407b7629df30d 2025-12-11 20:19:56,222 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b179915f407b7629df30d HTTP/1.1" 502 150 2025-12-11 20:19:56,223 - lava_callback - ERROR - Error processing callback for node 693b179915f407b7629df30d: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b179915f407b7629df30d (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b179915f407b7629df30d (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b179915f407b7629df30d (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:40136 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:19:57,475 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae86dfee8383e92b65929 2025-12-11 20:19:57,477 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae86dfee8383e92b65929 HTTP/1.1" 502 150 2025-12-11 20:19:57,477 - lava_callback - ERROR - Error processing callback for node 693ae86dfee8383e92b65929: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae86dfee8383e92b65929 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae86dfee8383e92b65929 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae86dfee8383e92b65929 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:20:00,071 - lava_callback - INFO - Received callback for node 693b0227fee8383e92b6ef77 INFO: 172.21.0.1:41722 - "POST /node/693b0227fee8383e92b6ef77 HTTP/1.0" 202 Accepted 2025-12-11 20:20:00,079 - lava_callback - INFO - Starting async job submit for node 693b0227fee8383e92b6ef77 2025-12-11 20:20:00,134 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:20:00,159 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0227fee8383e92b6ef77 HTTP/1.1" 502 150 2025-12-11 20:20:00,159 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0227fee8383e92b6ef77'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:00,160 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0227fee8383e92b6ef77 2025-12-11 20:20:00,160 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0227fee8383e92b6ef77 HTTP/1.1" 502 150 2025-12-11 20:20:00,160 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0227fee8383e92b6ef77'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:02,112 - lava_callback - INFO - Received callback for node 693b1be115f407b7629e030d INFO: 172.21.0.1:41730 - "POST /node/693b1be115f407b7629e030d HTTP/1.0" 202 Accepted 2025-12-11 20:20:02,121 - lava_callback - INFO - Starting async job submit for node 693b1be115f407b7629e030d 2025-12-11 20:20:02,164 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:20:02,164 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0227fee8383e92b6ef77 2025-12-11 20:20:02,165 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0227fee8383e92b6ef77 HTTP/1.1" 502 150 2025-12-11 20:20:02,165 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0227fee8383e92b6ef77'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:02,198 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1be115f407b7629e030d HTTP/1.1" 502 150 2025-12-11 20:20:02,198 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1be115f407b7629e030d'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:02,199 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1be115f407b7629e030d 2025-12-11 20:20:02,199 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1be115f407b7629e030d HTTP/1.1" 502 150 2025-12-11 20:20:02,199 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1be115f407b7629e030d'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:02,850 - lava_callback - INFO - Received callback for node 693b0ac3fee8383e92b71b1a INFO: 172.21.0.1:41738 - "POST /node/693b0ac3fee8383e92b71b1a HTTP/1.0" 202 Accepted 2025-12-11 20:20:02,859 - lava_callback - INFO - Starting async job submit for node 693b0ac3fee8383e92b71b1a 2025-12-11 20:20:02,917 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:20:02,930 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0ac3fee8383e92b71b1a HTTP/1.1" 502 150 2025-12-11 20:20:02,930 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0ac3fee8383e92b71b1a'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:02,930 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0ac3fee8383e92b71b1a 2025-12-11 20:20:02,931 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0ac3fee8383e92b71b1a HTTP/1.1" 502 150 2025-12-11 20:20:02,931 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0ac3fee8383e92b71b1a'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:04,200 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1be115f407b7629e030d 2025-12-11 20:20:04,201 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1be115f407b7629e030d HTTP/1.1" 502 150 2025-12-11 20:20:04,201 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1be115f407b7629e030d'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:04,931 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0ac3fee8383e92b71b1a 2025-12-11 20:20:04,932 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0ac3fee8383e92b71b1a HTTP/1.1" 502 150 2025-12-11 20:20:04,932 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0ac3fee8383e92b71b1a'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:06,165 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0227fee8383e92b6ef77 2025-12-11 20:20:06,166 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0227fee8383e92b6ef77 HTTP/1.1" 502 150 2025-12-11 20:20:06,166 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0227fee8383e92b6ef77'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:41744 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:20:07,349 - lava_callback - INFO - Received callback for node 693ae874fee8383e92b6594b INFO: 172.21.0.1:41760 - "POST /node/693ae874fee8383e92b6594b HTTP/1.0" 202 Accepted 2025-12-11 20:20:07,359 - lava_callback - INFO - Starting async job submit for node 693ae874fee8383e92b6594b 2025-12-11 20:20:07,464 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:20:07,478 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae874fee8383e92b6594b HTTP/1.1" 502 150 2025-12-11 20:20:07,478 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae874fee8383e92b6594b'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:07,478 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae874fee8383e92b6594b 2025-12-11 20:20:07,479 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae874fee8383e92b6594b HTTP/1.1" 502 150 2025-12-11 20:20:07,479 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae874fee8383e92b6594b'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:08,201 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1be115f407b7629e030d 2025-12-11 20:20:08,202 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1be115f407b7629e030d HTTP/1.1" 502 150 2025-12-11 20:20:08,203 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1be115f407b7629e030d'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:08,933 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0ac3fee8383e92b71b1a 2025-12-11 20:20:08,934 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0ac3fee8383e92b71b1a HTTP/1.1" 502 150 2025-12-11 20:20:08,934 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0ac3fee8383e92b71b1a'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:09,479 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae874fee8383e92b6594b 2025-12-11 20:20:09,480 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae874fee8383e92b6594b HTTP/1.1" 502 150 2025-12-11 20:20:09,481 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae874fee8383e92b6594b'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:13,481 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae874fee8383e92b6594b 2025-12-11 20:20:13,482 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae874fee8383e92b6594b HTTP/1.1" 502 150 2025-12-11 20:20:13,482 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae874fee8383e92b6594b'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:14,166 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0227fee8383e92b6ef77 2025-12-11 20:20:14,168 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0227fee8383e92b6ef77 HTTP/1.1" 502 150 2025-12-11 20:20:14,168 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0227fee8383e92b6ef77'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:56712 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 20:20:15,280 - lava_callback - INFO - Received callback for node 693b11e8fee8383e92b73a7f INFO: 172.21.0.1:56720 - "POST /node/693b11e8fee8383e92b73a7f HTTP/1.0" 202 Accepted 2025-12-11 20:20:15,288 - lava_callback - INFO - Starting async job submit for node 693b11e8fee8383e92b73a7f 2025-12-11 20:20:15,307 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:20:15,317 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b11e8fee8383e92b73a7f HTTP/1.1" 502 150 2025-12-11 20:20:15,317 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b11e8fee8383e92b73a7f'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:15,318 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b11e8fee8383e92b73a7f 2025-12-11 20:20:15,318 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b11e8fee8383e92b73a7f HTTP/1.1" 502 150 2025-12-11 20:20:15,318 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b11e8fee8383e92b73a7f'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:16,203 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1be115f407b7629e030d 2025-12-11 20:20:16,204 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1be115f407b7629e030d HTTP/1.1" 502 150 2025-12-11 20:20:16,204 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1be115f407b7629e030d'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:56730 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:20:16,934 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0ac3fee8383e92b71b1a 2025-12-11 20:20:16,936 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0ac3fee8383e92b71b1a HTTP/1.1" 502 150 2025-12-11 20:20:16,936 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0ac3fee8383e92b71b1a'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:17,319 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b11e8fee8383e92b73a7f 2025-12-11 20:20:17,320 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b11e8fee8383e92b73a7f HTTP/1.1" 502 150 2025-12-11 20:20:17,320 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b11e8fee8383e92b73a7f'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:21,320 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b11e8fee8383e92b73a7f 2025-12-11 20:20:21,321 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b11e8fee8383e92b73a7f HTTP/1.1" 502 150 2025-12-11 20:20:21,322 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b11e8fee8383e92b73a7f'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:21,482 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae874fee8383e92b6594b 2025-12-11 20:20:21,484 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae874fee8383e92b6594b HTTP/1.1" 502 150 2025-12-11 20:20:21,484 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae874fee8383e92b6594b'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:56492 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56506 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:20:29,322 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b11e8fee8383e92b73a7f 2025-12-11 20:20:29,323 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b11e8fee8383e92b73a7f HTTP/1.1" 502 150 2025-12-11 20:20:29,323 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b11e8fee8383e92b73a7f'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:30,168 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0227fee8383e92b6ef77 2025-12-11 20:20:30,169 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0227fee8383e92b6ef77 HTTP/1.1" 502 150 2025-12-11 20:20:30,170 - lava_callback - ERROR - Error processing callback for node 693b0227fee8383e92b6ef77: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0227fee8383e92b6ef77 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0227fee8383e92b6ef77 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0227fee8383e92b6ef77 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:20:32,204 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1be115f407b7629e030d 2025-12-11 20:20:32,206 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1be115f407b7629e030d HTTP/1.1" 502 150 2025-12-11 20:20:32,207 - lava_callback - ERROR - Error processing callback for node 693b1be115f407b7629e030d: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1be115f407b7629e030d (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1be115f407b7629e030d (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1be115f407b7629e030d (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:20:32,936 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0ac3fee8383e92b71b1a 2025-12-11 20:20:32,938 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0ac3fee8383e92b71b1a HTTP/1.1" 502 150 2025-12-11 20:20:32,938 - lava_callback - ERROR - Error processing callback for node 693b0ac3fee8383e92b71b1a: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0ac3fee8383e92b71b1a (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0ac3fee8383e92b71b1a (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0ac3fee8383e92b71b1a (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:20:33,584 - lava_callback - INFO - Received callback for node 693b1be315f407b7629e0310 INFO: 172.21.0.1:39624 - "POST /node/693b1be315f407b7629e0310 HTTP/1.0" 202 Accepted 2025-12-11 20:20:33,590 - lava_callback - INFO - Starting async job submit for node 693b1be315f407b7629e0310 2025-12-11 20:20:33,611 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:20:33,630 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1be315f407b7629e0310 HTTP/1.1" 502 150 2025-12-11 20:20:33,630 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1be315f407b7629e0310'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:33,630 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1be315f407b7629e0310 2025-12-11 20:20:33,630 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1be315f407b7629e0310 HTTP/1.1" 502 150 2025-12-11 20:20:33,630 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1be315f407b7629e0310'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:35,631 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1be315f407b7629e0310 2025-12-11 20:20:35,632 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1be315f407b7629e0310 HTTP/1.1" 502 150 2025-12-11 20:20:35,632 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1be315f407b7629e0310'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:39638 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:20:37,484 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae874fee8383e92b6594b 2025-12-11 20:20:37,485 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae874fee8383e92b6594b HTTP/1.1" 502 150 2025-12-11 20:20:37,485 - lava_callback - ERROR - Error processing callback for node 693ae874fee8383e92b6594b: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae874fee8383e92b6594b (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae874fee8383e92b6594b (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae874fee8383e92b6594b (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:20:39,632 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1be315f407b7629e0310 2025-12-11 20:20:39,633 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1be315f407b7629e0310 HTTP/1.1" 502 150 2025-12-11 20:20:39,633 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1be315f407b7629e0310'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:40,459 - lava_callback - INFO - Received callback for node 693b1bee15f407b7629e0320 INFO: 172.21.0.1:58968 - "POST /node/693b1bee15f407b7629e0320 HTTP/1.0" 202 Accepted 2025-12-11 20:20:40,472 - lava_callback - INFO - Starting async job submit for node 693b1bee15f407b7629e0320 2025-12-11 20:20:40,516 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:20:40,546 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bee15f407b7629e0320 HTTP/1.1" 502 150 2025-12-11 20:20:40,546 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bee15f407b7629e0320'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:40,547 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bee15f407b7629e0320 2025-12-11 20:20:40,547 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bee15f407b7629e0320 HTTP/1.1" 502 150 2025-12-11 20:20:40,547 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bee15f407b7629e0320'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:42,548 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bee15f407b7629e0320 2025-12-11 20:20:42,549 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bee15f407b7629e0320 HTTP/1.1" 502 150 2025-12-11 20:20:42,549 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bee15f407b7629e0320'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:45,323 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b11e8fee8383e92b73a7f 2025-12-11 20:20:45,327 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b11e8fee8383e92b73a7f HTTP/1.1" 502 150 2025-12-11 20:20:45,327 - lava_callback - ERROR - Error processing callback for node 693b11e8fee8383e92b73a7f: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b11e8fee8383e92b73a7f (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b11e8fee8383e92b73a7f (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b11e8fee8383e92b73a7f (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:20:46,306 - lava_callback - INFO - Received callback for node 693b1472fee8383e92b743e2 INFO: 172.21.0.1:58972 - "POST /node/693b1472fee8383e92b743e2 HTTP/1.0" 202 Accepted 2025-12-11 20:20:46,313 - lava_callback - INFO - Starting async job submit for node 693b1472fee8383e92b743e2 2025-12-11 20:20:46,348 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:20:46,380 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1472fee8383e92b743e2 HTTP/1.1" 502 150 2025-12-11 20:20:46,380 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1472fee8383e92b743e2'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:46,380 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1472fee8383e92b743e2 2025-12-11 20:20:46,381 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1472fee8383e92b743e2 HTTP/1.1" 502 150 2025-12-11 20:20:46,381 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1472fee8383e92b743e2'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:58980 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:20:46,549 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bee15f407b7629e0320 2025-12-11 20:20:46,550 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bee15f407b7629e0320 HTTP/1.1" 502 150 2025-12-11 20:20:46,550 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bee15f407b7629e0320'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:47,634 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1be315f407b7629e0310 2025-12-11 20:20:47,635 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1be315f407b7629e0310 HTTP/1.1" 502 150 2025-12-11 20:20:47,635 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1be315f407b7629e0310'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:47,765 - lava_callback - INFO - Received callback for node 693b18d315f407b7629df558 INFO: 172.21.0.1:58992 - "POST /node/693b18d315f407b7629df558 HTTP/1.0" 202 Accepted 2025-12-11 20:20:47,772 - lava_callback - INFO - Starting async job submit for node 693b18d315f407b7629df558 2025-12-11 20:20:47,806 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:20:47,817 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18d315f407b7629df558 HTTP/1.1" 502 150 2025-12-11 20:20:47,817 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18d315f407b7629df558'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:47,817 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18d315f407b7629df558 2025-12-11 20:20:47,818 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18d315f407b7629df558 HTTP/1.1" 502 150 2025-12-11 20:20:47,818 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18d315f407b7629df558'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:48,381 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1472fee8383e92b743e2 2025-12-11 20:20:48,382 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1472fee8383e92b743e2 HTTP/1.1" 502 150 2025-12-11 20:20:48,382 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1472fee8383e92b743e2'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:48,699 - lava_callback - INFO - Received callback for node 693aefc0fee8383e92b68ca4 INFO: 172.21.0.1:59008 - "POST /node/693aefc0fee8383e92b68ca4 HTTP/1.0" 202 Accepted 2025-12-11 20:20:48,705 - lava_callback - INFO - Starting async job submit for node 693aefc0fee8383e92b68ca4 2025-12-11 20:20:48,729 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:20:48,739 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aefc0fee8383e92b68ca4 HTTP/1.1" 502 150 2025-12-11 20:20:48,739 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aefc0fee8383e92b68ca4'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:48,739 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aefc0fee8383e92b68ca4 2025-12-11 20:20:48,740 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aefc0fee8383e92b68ca4 HTTP/1.1" 502 150 2025-12-11 20:20:48,740 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aefc0fee8383e92b68ca4'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:49,818 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18d315f407b7629df558 2025-12-11 20:20:49,819 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18d315f407b7629df558 HTTP/1.1" 502 150 2025-12-11 20:20:49,819 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18d315f407b7629df558'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:50,740 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aefc0fee8383e92b68ca4 2025-12-11 20:20:50,742 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aefc0fee8383e92b68ca4 HTTP/1.1" 502 150 2025-12-11 20:20:50,742 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aefc0fee8383e92b68ca4'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:52,383 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1472fee8383e92b743e2 2025-12-11 20:20:52,384 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1472fee8383e92b743e2 HTTP/1.1" 502 150 2025-12-11 20:20:52,384 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1472fee8383e92b743e2'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:53,820 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18d315f407b7629df558 2025-12-11 20:20:53,821 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18d315f407b7629df558 HTTP/1.1" 502 150 2025-12-11 20:20:53,821 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18d315f407b7629df558'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:54,551 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bee15f407b7629e0320 2025-12-11 20:20:54,552 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bee15f407b7629e0320 HTTP/1.1" 502 150 2025-12-11 20:20:54,552 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bee15f407b7629e0320'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:20:54,742 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aefc0fee8383e92b68ca4 2025-12-11 20:20:54,743 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aefc0fee8383e92b68ca4 HTTP/1.1" 502 150 2025-12-11 20:20:54,743 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aefc0fee8383e92b68ca4'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:33324 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:21:00,384 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1472fee8383e92b743e2 2025-12-11 20:21:00,386 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1472fee8383e92b743e2 HTTP/1.1" 502 150 2025-12-11 20:21:00,386 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1472fee8383e92b743e2'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:21:01,821 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18d315f407b7629df558 2025-12-11 20:21:01,822 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18d315f407b7629df558 HTTP/1.1" 502 150 2025-12-11 20:21:01,822 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18d315f407b7629df558'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:21:02,744 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aefc0fee8383e92b68ca4 2025-12-11 20:21:02,745 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aefc0fee8383e92b68ca4 HTTP/1.1" 502 150 2025-12-11 20:21:02,745 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aefc0fee8383e92b68ca4'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:21:03,635 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1be315f407b7629e0310 2025-12-11 20:21:03,638 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1be315f407b7629e0310 HTTP/1.1" 502 150 2025-12-11 20:21:03,638 - lava_callback - ERROR - Error processing callback for node 693b1be315f407b7629e0310: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1be315f407b7629e0310 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1be315f407b7629e0310 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1be315f407b7629e0310 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:42668 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:21:06,534 - lava_callback - INFO - Received callback for node 693ae876fee8383e92b65952 INFO: 172.21.0.1:42676 - "POST /node/693ae876fee8383e92b65952 HTTP/1.0" 202 Accepted 2025-12-11 20:21:06,541 - lava_callback - INFO - Starting async job submit for node 693ae876fee8383e92b65952 2025-12-11 20:21:06,618 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:21:06,631 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae876fee8383e92b65952 HTTP/1.1" 502 150 2025-12-11 20:21:06,631 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae876fee8383e92b65952'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:21:06,631 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae876fee8383e92b65952 2025-12-11 20:21:06,632 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae876fee8383e92b65952 HTTP/1.1" 502 150 2025-12-11 20:21:06,632 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae876fee8383e92b65952'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:21:07,132 - lava_callback - INFO - Received callback for node 693ae877fee8383e92b6595e INFO: 172.21.0.1:42684 - "POST /node/693ae877fee8383e92b6595e HTTP/1.0" 202 Accepted 2025-12-11 20:21:07,138 - lava_callback - INFO - Starting async job submit for node 693ae877fee8383e92b6595e 2025-12-11 20:21:07,172 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:21:07,181 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae877fee8383e92b6595e HTTP/1.1" 502 150 2025-12-11 20:21:07,181 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae877fee8383e92b6595e'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:21:07,181 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae877fee8383e92b6595e 2025-12-11 20:21:07,182 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae877fee8383e92b6595e HTTP/1.1" 502 150 2025-12-11 20:21:07,182 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae877fee8383e92b6595e'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:21:08,177 - lava_callback - INFO - Received callback for node 693af790fee8383e92b6c0f8 INFO: 172.21.0.1:42700 - "POST /node/693af790fee8383e92b6c0f8 HTTP/1.0" 202 Accepted 2025-12-11 20:21:08,183 - lava_callback - INFO - Starting async job submit for node 693af790fee8383e92b6c0f8 2025-12-11 20:21:08,201 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:21:08,235 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af790fee8383e92b6c0f8 HTTP/1.1" 502 150 2025-12-11 20:21:08,236 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af790fee8383e92b6c0f8'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:21:08,236 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af790fee8383e92b6c0f8 2025-12-11 20:21:08,236 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af790fee8383e92b6c0f8 HTTP/1.1" 502 150 2025-12-11 20:21:08,236 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af790fee8383e92b6c0f8'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:21:08,632 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae876fee8383e92b65952 2025-12-11 20:21:08,633 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae876fee8383e92b65952 HTTP/1.1" 502 150 2025-12-11 20:21:08,634 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae876fee8383e92b65952'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:21:09,182 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae877fee8383e92b6595e 2025-12-11 20:21:09,184 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae877fee8383e92b6595e HTTP/1.1" 502 150 2025-12-11 20:21:09,184 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae877fee8383e92b6595e'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:21:09,600 - lava_callback - INFO - Received callback for node 693b18d415f407b7629df55a INFO: 172.21.0.1:42704 - "POST /node/693b18d415f407b7629df55a HTTP/1.0" 202 Accepted 2025-12-11 20:21:09,606 - lava_callback - INFO - Starting async job submit for node 693b18d415f407b7629df55a 2025-12-11 20:21:09,628 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:21:09,641 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18d415f407b7629df55a HTTP/1.1" 502 150 2025-12-11 20:21:09,641 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18d415f407b7629df55a'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:21:09,641 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18d415f407b7629df55a 2025-12-11 20:21:09,641 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18d415f407b7629df55a HTTP/1.1" 502 150 2025-12-11 20:21:09,641 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18d415f407b7629df55a'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:21:10,237 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af790fee8383e92b6c0f8 2025-12-11 20:21:10,238 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af790fee8383e92b6c0f8 HTTP/1.1" 502 150 2025-12-11 20:21:10,238 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af790fee8383e92b6c0f8'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:21:10,552 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bee15f407b7629e0320 2025-12-11 20:21:10,553 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bee15f407b7629e0320 HTTP/1.1" 502 150 2025-12-11 20:21:10,554 - lava_callback - ERROR - Error processing callback for node 693b1bee15f407b7629e0320: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1bee15f407b7629e0320 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1bee15f407b7629e0320 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1bee15f407b7629e0320 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:21:11,642 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18d415f407b7629df55a 2025-12-11 20:21:11,643 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18d415f407b7629df55a HTTP/1.1" 502 150 2025-12-11 20:21:11,643 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18d415f407b7629df55a'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:21:12,634 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae876fee8383e92b65952 2025-12-11 20:21:12,635 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae876fee8383e92b65952 HTTP/1.1" 502 150 2025-12-11 20:21:12,635 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae876fee8383e92b65952'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:21:13,184 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae877fee8383e92b6595e 2025-12-11 20:21:13,185 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae877fee8383e92b6595e HTTP/1.1" 502 150 2025-12-11 20:21:13,185 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae877fee8383e92b6595e'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:21:14,238 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af790fee8383e92b6c0f8 2025-12-11 20:21:14,239 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af790fee8383e92b6c0f8 HTTP/1.1" 502 150 2025-12-11 20:21:14,239 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af790fee8383e92b6c0f8'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:39990 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 20:21:15,643 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18d415f407b7629df55a 2025-12-11 20:21:15,644 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18d415f407b7629df55a HTTP/1.1" 502 150 2025-12-11 20:21:15,644 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18d415f407b7629df55a'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:21:16,386 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1472fee8383e92b743e2 2025-12-11 20:21:16,388 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1472fee8383e92b743e2 HTTP/1.1" 502 150 2025-12-11 20:21:16,388 - lava_callback - ERROR - Error processing callback for node 693b1472fee8383e92b743e2: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1472fee8383e92b743e2 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1472fee8383e92b743e2 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1472fee8383e92b743e2 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:39992 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:21:17,822 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18d315f407b7629df558 2025-12-11 20:21:17,824 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18d315f407b7629df558 HTTP/1.1" 502 150 2025-12-11 20:21:17,824 - lava_callback - ERROR - Error processing callback for node 693b18d315f407b7629df558: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b18d315f407b7629df558 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b18d315f407b7629df558 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b18d315f407b7629df558 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:21:18,745 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aefc0fee8383e92b68ca4 2025-12-11 20:21:18,747 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aefc0fee8383e92b68ca4 HTTP/1.1" 502 150 2025-12-11 20:21:18,747 - lava_callback - ERROR - Error processing callback for node 693aefc0fee8383e92b68ca4: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aefc0fee8383e92b68ca4 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aefc0fee8383e92b68ca4 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aefc0fee8383e92b68ca4 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:21:20,635 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae876fee8383e92b65952 2025-12-11 20:21:20,637 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae876fee8383e92b65952 HTTP/1.1" 502 150 2025-12-11 20:21:20,637 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae876fee8383e92b65952'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:21:21,186 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae877fee8383e92b6595e 2025-12-11 20:21:21,187 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae877fee8383e92b6595e HTTP/1.1" 502 150 2025-12-11 20:21:21,187 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae877fee8383e92b6595e'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:21:22,240 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af790fee8383e92b6c0f8 2025-12-11 20:21:22,241 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af790fee8383e92b6c0f8 HTTP/1.1" 502 150 2025-12-11 20:21:22,241 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af790fee8383e92b6c0f8'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:21:23,645 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18d415f407b7629df55a 2025-12-11 20:21:23,646 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18d415f407b7629df55a HTTP/1.1" 502 150 2025-12-11 20:21:23,646 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18d415f407b7629df55a'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:40824 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40832 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59270 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:21:36,637 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae876fee8383e92b65952 2025-12-11 20:21:36,638 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae876fee8383e92b65952 HTTP/1.1" 502 150 2025-12-11 20:21:36,638 - lava_callback - ERROR - Error processing callback for node 693ae876fee8383e92b65952: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae876fee8383e92b65952 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae876fee8383e92b65952 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae876fee8383e92b65952 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:21:37,187 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae877fee8383e92b6595e 2025-12-11 20:21:37,192 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae877fee8383e92b6595e HTTP/1.1" 502 150 2025-12-11 20:21:37,192 - lava_callback - ERROR - Error processing callback for node 693ae877fee8383e92b6595e: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae877fee8383e92b6595e (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae877fee8383e92b6595e (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae877fee8383e92b6595e (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:21:38,241 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af790fee8383e92b6c0f8 2025-12-11 20:21:38,243 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af790fee8383e92b6c0f8 HTTP/1.1" 502 150 2025-12-11 20:21:38,243 - lava_callback - ERROR - Error processing callback for node 693af790fee8383e92b6c0f8: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af790fee8383e92b6c0f8 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af790fee8383e92b6c0f8 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af790fee8383e92b6c0f8 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:21:39,646 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18d415f407b7629df55a 2025-12-11 20:21:39,647 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18d415f407b7629df55a HTTP/1.1" 502 150 2025-12-11 20:21:39,648 - lava_callback - ERROR - Error processing callback for node 693b18d415f407b7629df55a: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b18d415f407b7629df55a (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b18d415f407b7629df55a (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b18d415f407b7629df55a (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:41912 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55338 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:22:06,268 - lava_callback - INFO - Received callback for node 693af4f8fee8383e92b6b0f9 INFO: 172.21.0.1:54390 - "POST /node/693af4f8fee8383e92b6b0f9 HTTP/1.0" 202 Accepted 2025-12-11 20:22:06,276 - lava_callback - INFO - Starting async job submit for node 693af4f8fee8383e92b6b0f9 2025-12-11 20:22:06,317 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:22:06,328 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af4f8fee8383e92b6b0f9 HTTP/1.1" 502 150 2025-12-11 20:22:06,328 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af4f8fee8383e92b6b0f9'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:22:06,329 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af4f8fee8383e92b6b0f9 2025-12-11 20:22:06,329 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af4f8fee8383e92b6b0f9 HTTP/1.1" 502 150 2025-12-11 20:22:06,329 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af4f8fee8383e92b6b0f9'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:54398 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:22:08,330 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af4f8fee8383e92b6b0f9 2025-12-11 20:22:08,331 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af4f8fee8383e92b6b0f9 HTTP/1.1" 502 150 2025-12-11 20:22:08,331 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af4f8fee8383e92b6b0f9'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:22:12,331 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af4f8fee8383e92b6b0f9 2025-12-11 20:22:12,332 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af4f8fee8383e92b6b0f9 HTTP/1.1" 502 150 2025-12-11 20:22:12,332 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af4f8fee8383e92b6b0f9'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:40204 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:40206 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:22:17,980 - lava_callback - INFO - Received callback for node 693b1c5b15f407b7629e04dc INFO: 172.21.0.1:40208 - "POST /node/693b1c5b15f407b7629e04dc HTTP/1.0" 202 Accepted 2025-12-11 20:22:17,986 - lava_callback - INFO - Starting async job submit for node 693b1c5b15f407b7629e04dc 2025-12-11 20:22:18,011 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:22:18,023 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c5b15f407b7629e04dc HTTP/1.1" 502 150 2025-12-11 20:22:18,023 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c5b15f407b7629e04dc'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:22:18,023 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c5b15f407b7629e04dc 2025-12-11 20:22:18,024 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c5b15f407b7629e04dc HTTP/1.1" 502 150 2025-12-11 20:22:18,024 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c5b15f407b7629e04dc'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:22:20,025 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c5b15f407b7629e04dc 2025-12-11 20:22:20,026 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c5b15f407b7629e04dc HTTP/1.1" 502 150 2025-12-11 20:22:20,026 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c5b15f407b7629e04dc'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:22:20,333 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af4f8fee8383e92b6b0f9 2025-12-11 20:22:20,334 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af4f8fee8383e92b6b0f9 HTTP/1.1" 502 150 2025-12-11 20:22:20,334 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af4f8fee8383e92b6b0f9'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:22:24,026 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c5b15f407b7629e04dc 2025-12-11 20:22:24,027 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c5b15f407b7629e04dc HTTP/1.1" 502 150 2025-12-11 20:22:24,027 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c5b15f407b7629e04dc'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:58618 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58620 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:22:27,726 - lava_callback - INFO - Received callback for node 693af05afee8383e92b6912d INFO: 172.21.0.1:58632 - "POST /node/693af05afee8383e92b6912d HTTP/1.0" 202 Accepted 2025-12-11 20:22:27,732 - lava_callback - INFO - Starting async job submit for node 693af05afee8383e92b6912d 2025-12-11 20:22:27,753 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:22:27,763 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af05afee8383e92b6912d HTTP/1.1" 502 150 2025-12-11 20:22:27,764 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af05afee8383e92b6912d'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:22:27,764 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af05afee8383e92b6912d 2025-12-11 20:22:27,764 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af05afee8383e92b6912d HTTP/1.1" 502 150 2025-12-11 20:22:27,764 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af05afee8383e92b6912d'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:22:29,765 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af05afee8383e92b6912d 2025-12-11 20:22:29,766 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af05afee8383e92b6912d HTTP/1.1" 502 150 2025-12-11 20:22:29,766 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af05afee8383e92b6912d'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:22:32,028 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c5b15f407b7629e04dc 2025-12-11 20:22:32,029 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c5b15f407b7629e04dc HTTP/1.1" 502 150 2025-12-11 20:22:32,029 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c5b15f407b7629e04dc'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:22:33,766 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af05afee8383e92b6912d 2025-12-11 20:22:33,767 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af05afee8383e92b6912d HTTP/1.1" 502 150 2025-12-11 20:22:33,768 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af05afee8383e92b6912d'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:22:36,334 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af4f8fee8383e92b6b0f9 2025-12-11 20:22:36,335 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af4f8fee8383e92b6b0f9 HTTP/1.1" 502 150 2025-12-11 20:22:36,336 - lava_callback - ERROR - Error processing callback for node 693af4f8fee8383e92b6b0f9: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af4f8fee8383e92b6b0f9 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af4f8fee8383e92b6b0f9 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af4f8fee8383e92b6b0f9 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:49252 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:22:37,389 - lava_callback - INFO - Received callback for node 693b1cba15f407b7629e0656 INFO: 172.21.0.1:49264 - "POST /node/693b1cba15f407b7629e0656 HTTP/1.0" 202 Accepted 2025-12-11 20:22:37,396 - lava_callback - INFO - Starting async job submit for node 693b1cba15f407b7629e0656 2025-12-11 20:22:37,434 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:22:37,444 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1cba15f407b7629e0656 HTTP/1.1" 502 150 2025-12-11 20:22:37,444 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1cba15f407b7629e0656'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:22:37,444 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1cba15f407b7629e0656 2025-12-11 20:22:37,445 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1cba15f407b7629e0656 HTTP/1.1" 502 150 2025-12-11 20:22:37,445 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1cba15f407b7629e0656'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:22:39,445 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1cba15f407b7629e0656 2025-12-11 20:22:39,447 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1cba15f407b7629e0656 HTTP/1.1" 502 150 2025-12-11 20:22:39,447 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1cba15f407b7629e0656'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:22:41,768 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af05afee8383e92b6912d 2025-12-11 20:22:41,769 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af05afee8383e92b6912d HTTP/1.1" 502 150 2025-12-11 20:22:41,769 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af05afee8383e92b6912d'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:22:43,447 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1cba15f407b7629e0656 2025-12-11 20:22:43,448 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1cba15f407b7629e0656 HTTP/1.1" 502 150 2025-12-11 20:22:43,448 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1cba15f407b7629e0656'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:22:46,163 - lava_callback - INFO - Received callback for node 693b18d415f407b7629df55e INFO: 172.21.0.1:59762 - "POST /node/693b18d415f407b7629df55e HTTP/1.0" 202 Accepted 2025-12-11 20:22:46,173 - lava_callback - INFO - Starting async job submit for node 693b18d415f407b7629df55e 2025-12-11 20:22:46,213 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:22:46,230 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18d415f407b7629df55e HTTP/1.1" 502 150 2025-12-11 20:22:46,230 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18d415f407b7629df55e'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:22:46,230 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18d415f407b7629df55e 2025-12-11 20:22:46,231 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18d415f407b7629df55e HTTP/1.1" 502 150 2025-12-11 20:22:46,231 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18d415f407b7629df55e'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:59764 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:22:48,030 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c5b15f407b7629e04dc 2025-12-11 20:22:48,032 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c5b15f407b7629e04dc HTTP/1.1" 502 150 2025-12-11 20:22:48,032 - lava_callback - ERROR - Error processing callback for node 693b1c5b15f407b7629e04dc: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1c5b15f407b7629e04dc (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1c5b15f407b7629e04dc (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1c5b15f407b7629e04dc (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:22:48,232 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18d415f407b7629df55e 2025-12-11 20:22:48,233 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18d415f407b7629df55e HTTP/1.1" 502 150 2025-12-11 20:22:48,233 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18d415f407b7629df55e'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:22:51,449 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1cba15f407b7629e0656 2025-12-11 20:22:51,450 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1cba15f407b7629e0656 HTTP/1.1" 502 150 2025-12-11 20:22:51,450 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1cba15f407b7629e0656'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:22:52,233 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18d415f407b7629df55e 2025-12-11 20:22:52,234 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18d415f407b7629df55e HTTP/1.1" 502 150 2025-12-11 20:22:52,234 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18d415f407b7629df55e'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:51128 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:22:57,769 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af05afee8383e92b6912d 2025-12-11 20:22:57,771 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af05afee8383e92b6912d HTTP/1.1" 502 150 2025-12-11 20:22:57,771 - lava_callback - ERROR - Error processing callback for node 693af05afee8383e92b6912d: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af05afee8383e92b6912d (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af05afee8383e92b6912d (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af05afee8383e92b6912d (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:23:00,235 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18d415f407b7629df55e 2025-12-11 20:23:00,236 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18d415f407b7629df55e HTTP/1.1" 502 150 2025-12-11 20:23:00,236 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b18d415f407b7629df55e'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:53270 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:23:07,450 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1cba15f407b7629e0656 2025-12-11 20:23:07,451 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1cba15f407b7629e0656 HTTP/1.1" 502 150 2025-12-11 20:23:07,451 - lava_callback - ERROR - Error processing callback for node 693b1cba15f407b7629e0656: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1cba15f407b7629e0656 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1cba15f407b7629e0656 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1cba15f407b7629e0656 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:23:11,488 - lava_callback - INFO - Received callback for node 693b123efee8383e92b73bfe INFO: 172.21.0.1:59216 - "POST /node/693b123efee8383e92b73bfe HTTP/1.0" 202 Accepted 2025-12-11 20:23:11,500 - lava_callback - INFO - Starting async job submit for node 693b123efee8383e92b73bfe 2025-12-11 20:23:11,521 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:23:11,558 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b123efee8383e92b73bfe HTTP/1.1" 502 150 2025-12-11 20:23:11,559 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b123efee8383e92b73bfe'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:23:11,559 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b123efee8383e92b73bfe 2025-12-11 20:23:11,559 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b123efee8383e92b73bfe HTTP/1.1" 502 150 2025-12-11 20:23:11,559 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b123efee8383e92b73bfe'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:23:13,560 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b123efee8383e92b73bfe 2025-12-11 20:23:13,561 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b123efee8383e92b73bfe HTTP/1.1" 502 150 2025-12-11 20:23:13,561 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b123efee8383e92b73bfe'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:59232 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 20:23:16,236 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b18d415f407b7629df55e 2025-12-11 20:23:16,237 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b18d415f407b7629df55e HTTP/1.1" 502 150 2025-12-11 20:23:16,238 - lava_callback - ERROR - Error processing callback for node 693b18d415f407b7629df55e: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b18d415f407b7629df55e (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b18d415f407b7629df55e (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b18d415f407b7629df55e (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:59240 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:23:17,561 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b123efee8383e92b73bfe 2025-12-11 20:23:17,563 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b123efee8383e92b73bfe HTTP/1.1" 502 150 2025-12-11 20:23:17,563 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b123efee8383e92b73bfe'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:23:23,694 - lava_callback - INFO - Received callback for node 6938852763d980a974804841 INFO: 172.21.0.1:59684 - "POST /node/6938852763d980a974804841 HTTP/1.0" 202 Accepted 2025-12-11 20:23:23,699 - lava_callback - INFO - Starting async job submit for node 6938852763d980a974804841 2025-12-11 20:23:23,700 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:23:23,710 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938852763d980a974804841 HTTP/1.1" 502 150 2025-12-11 20:23:23,710 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938852763d980a974804841'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:23:23,710 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938852763d980a974804841 2025-12-11 20:23:23,711 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938852763d980a974804841 HTTP/1.1" 502 150 2025-12-11 20:23:23,711 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938852763d980a974804841'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:23:23,889 - lava_callback - INFO - Received callback for node 6938853663d980a97480484d INFO: 172.21.0.1:59698 - "POST /node/6938853663d980a97480484d HTTP/1.0" 202 Accepted 2025-12-11 20:23:23,897 - lava_callback - INFO - Starting async job submit for node 6938853663d980a97480484d 2025-12-11 20:23:23,898 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:23:23,909 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938853663d980a97480484d HTTP/1.1" 502 150 2025-12-11 20:23:23,909 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938853663d980a97480484d'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:23:23,909 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938853663d980a97480484d 2025-12-11 20:23:23,910 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938853663d980a97480484d HTTP/1.1" 502 150 2025-12-11 20:23:23,910 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938853663d980a97480484d'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:23:24,088 - lava_callback - INFO - Received callback for node 6938853a63d980a974804852 INFO: 172.21.0.1:59702 - "POST /node/6938853a63d980a974804852 HTTP/1.0" 202 Accepted 2025-12-11 20:23:24,095 - lava_callback - INFO - Starting async job submit for node 6938853a63d980a974804852 2025-12-11 20:23:24,096 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:23:24,118 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938853a63d980a974804852 HTTP/1.1" 502 150 2025-12-11 20:23:24,118 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938853a63d980a974804852'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:23:24,119 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938853a63d980a974804852 2025-12-11 20:23:24,119 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938853a63d980a974804852 HTTP/1.1" 502 150 2025-12-11 20:23:24,119 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938853a63d980a974804852'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:23:25,563 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b123efee8383e92b73bfe 2025-12-11 20:23:25,564 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b123efee8383e92b73bfe HTTP/1.1" 502 150 2025-12-11 20:23:25,564 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b123efee8383e92b73bfe'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:23:25,711 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938852763d980a974804841 2025-12-11 20:23:25,713 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938852763d980a974804841 HTTP/1.1" 502 150 2025-12-11 20:23:25,713 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938852763d980a974804841'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:23:25,910 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938853663d980a97480484d 2025-12-11 20:23:25,911 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938853663d980a97480484d HTTP/1.1" 502 150 2025-12-11 20:23:25,911 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938853663d980a97480484d'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:23:26,120 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938853a63d980a974804852 2025-12-11 20:23:26,121 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938853a63d980a974804852 HTTP/1.1" 502 150 2025-12-11 20:23:26,121 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938853a63d980a974804852'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:23:26,462 - lava_callback - INFO - Received callback for node 693b0e14fee8383e92b72496 INFO: 172.21.0.1:59704 - "POST /node/693b0e14fee8383e92b72496 HTTP/1.0" 202 Accepted 2025-12-11 20:23:26,469 - lava_callback - INFO - Starting async job submit for node 693b0e14fee8383e92b72496 INFO: 172.21.0.1:59718 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:23:26,556 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:23:26,566 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e14fee8383e92b72496 HTTP/1.1" 502 150 2025-12-11 20:23:26,566 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e14fee8383e92b72496'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:23:26,566 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e14fee8383e92b72496 2025-12-11 20:23:26,567 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e14fee8383e92b72496 HTTP/1.1" 502 150 2025-12-11 20:23:26,567 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e14fee8383e92b72496'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:59728 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:23:28,567 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e14fee8383e92b72496 2025-12-11 20:23:28,568 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e14fee8383e92b72496 HTTP/1.1" 502 150 2025-12-11 20:23:28,568 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e14fee8383e92b72496'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:23:29,713 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938852763d980a974804841 2025-12-11 20:23:29,714 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938852763d980a974804841 HTTP/1.1" 502 150 2025-12-11 20:23:29,714 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938852763d980a974804841'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:23:29,911 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938853663d980a97480484d 2025-12-11 20:23:29,912 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938853663d980a97480484d HTTP/1.1" 502 150 2025-12-11 20:23:29,913 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938853663d980a97480484d'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:23:30,121 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938853a63d980a974804852 2025-12-11 20:23:30,122 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938853a63d980a974804852 HTTP/1.1" 502 150 2025-12-11 20:23:30,122 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938853a63d980a974804852'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:23:32,569 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e14fee8383e92b72496 2025-12-11 20:23:32,570 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e14fee8383e92b72496 HTTP/1.1" 502 150 2025-12-11 20:23:32,570 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e14fee8383e92b72496'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:23:35,992 - lava_callback - INFO - Received callback for node 693b0c18fee8383e92b71ef8 INFO: 172.21.0.1:54420 - "POST /node/693b0c18fee8383e92b71ef8 HTTP/1.0" 202 Accepted 2025-12-11 20:23:36,001 - lava_callback - INFO - Starting async job submit for node 693b0c18fee8383e92b71ef8 2025-12-11 20:23:36,059 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:23:36,076 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0c18fee8383e92b71ef8 HTTP/1.1" 502 150 2025-12-11 20:23:36,076 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0c18fee8383e92b71ef8'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:23:36,077 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0c18fee8383e92b71ef8 2025-12-11 20:23:36,077 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0c18fee8383e92b71ef8 HTTP/1.1" 502 150 2025-12-11 20:23:36,078 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0c18fee8383e92b71ef8'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:54428 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:23:37,714 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938852763d980a974804841 2025-12-11 20:23:37,716 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938852763d980a974804841 HTTP/1.1" 502 150 2025-12-11 20:23:37,716 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938852763d980a974804841'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:23:37,913 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938853663d980a97480484d 2025-12-11 20:23:37,914 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938853663d980a97480484d HTTP/1.1" 502 150 2025-12-11 20:23:37,914 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938853663d980a97480484d'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:23:38,078 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0c18fee8383e92b71ef8 2025-12-11 20:23:38,079 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0c18fee8383e92b71ef8 HTTP/1.1" 502 150 2025-12-11 20:23:38,079 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0c18fee8383e92b71ef8'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:23:38,122 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938853a63d980a974804852 2025-12-11 20:23:38,123 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938853a63d980a974804852 HTTP/1.1" 502 150 2025-12-11 20:23:38,123 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938853a63d980a974804852'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:23:40,570 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e14fee8383e92b72496 2025-12-11 20:23:40,572 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e14fee8383e92b72496 HTTP/1.1" 502 150 2025-12-11 20:23:40,572 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e14fee8383e92b72496'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:23:41,565 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b123efee8383e92b73bfe 2025-12-11 20:23:41,566 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b123efee8383e92b73bfe HTTP/1.1" 502 150 2025-12-11 20:23:41,566 - lava_callback - ERROR - Error processing callback for node 693b123efee8383e92b73bfe: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b123efee8383e92b73bfe (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b123efee8383e92b73bfe (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b123efee8383e92b73bfe (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:23:42,079 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0c18fee8383e92b71ef8 2025-12-11 20:23:42,081 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0c18fee8383e92b71ef8 HTTP/1.1" 502 150 2025-12-11 20:23:42,081 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0c18fee8383e92b71ef8'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:41776 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:23:50,081 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0c18fee8383e92b71ef8 2025-12-11 20:23:50,083 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0c18fee8383e92b71ef8 HTTP/1.1" 502 150 2025-12-11 20:23:50,083 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0c18fee8383e92b71ef8'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:23:53,716 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938852763d980a974804841 2025-12-11 20:23:53,717 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938852763d980a974804841 HTTP/1.1" 502 150 2025-12-11 20:23:53,717 - lava_callback - ERROR - Error processing callback for node 6938852763d980a974804841: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938852763d980a974804841 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938852763d980a974804841 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938852763d980a974804841 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:23:53,914 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938853663d980a97480484d 2025-12-11 20:23:53,916 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938853663d980a97480484d HTTP/1.1" 502 150 2025-12-11 20:23:53,916 - lava_callback - ERROR - Error processing callback for node 6938853663d980a97480484d: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938853663d980a97480484d (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938853663d980a97480484d (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938853663d980a97480484d (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:23:54,123 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938853a63d980a974804852 2025-12-11 20:23:54,125 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938853a63d980a974804852 HTTP/1.1" 502 150 2025-12-11 20:23:54,125 - lava_callback - ERROR - Error processing callback for node 6938853a63d980a974804852: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938853a63d980a974804852 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938853a63d980a974804852 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938853a63d980a974804852 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:57752 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:23:56,572 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e14fee8383e92b72496 2025-12-11 20:23:56,573 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e14fee8383e92b72496 HTTP/1.1" 502 150 2025-12-11 20:23:56,573 - lava_callback - ERROR - Error processing callback for node 693b0e14fee8383e92b72496: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e14fee8383e92b72496 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e14fee8383e92b72496 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e14fee8383e92b72496 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:24:04,804 - lava_callback - INFO - Received callback for node 693b1c5b15f407b7629e04da INFO: 172.21.0.1:48240 - "POST /node/693b1c5b15f407b7629e04da HTTP/1.0" 202 Accepted 2025-12-11 20:24:04,812 - lava_callback - INFO - Starting async job submit for node 693b1c5b15f407b7629e04da 2025-12-11 20:24:04,854 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:24:04,863 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c5b15f407b7629e04da HTTP/1.1" 502 150 2025-12-11 20:24:04,863 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c5b15f407b7629e04da'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:04,864 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c5b15f407b7629e04da 2025-12-11 20:24:04,864 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c5b15f407b7629e04da HTTP/1.1" 502 150 2025-12-11 20:24:04,864 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c5b15f407b7629e04da'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:06,083 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0c18fee8383e92b71ef8 2025-12-11 20:24:06,084 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0c18fee8383e92b71ef8 HTTP/1.1" 502 150 2025-12-11 20:24:06,085 - lava_callback - ERROR - Error processing callback for node 693b0c18fee8383e92b71ef8: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0c18fee8383e92b71ef8 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0c18fee8383e92b71ef8 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0c18fee8383e92b71ef8 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:24:06,323 - lava_callback - INFO - Received callback for node 693af7b5fee8383e92b6c180 INFO: 172.21.0.1:48248 - "POST /node/693af7b5fee8383e92b6c180 HTTP/1.0" 202 Accepted 2025-12-11 20:24:06,329 - lava_callback - INFO - Starting async job submit for node 693af7b5fee8383e92b6c180 2025-12-11 20:24:06,347 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:24:06,360 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af7b5fee8383e92b6c180 HTTP/1.1" 502 150 2025-12-11 20:24:06,360 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af7b5fee8383e92b6c180'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:06,360 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af7b5fee8383e92b6c180 2025-12-11 20:24:06,361 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af7b5fee8383e92b6c180 HTTP/1.1" 502 150 2025-12-11 20:24:06,361 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af7b5fee8383e92b6c180'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:48250 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:24:06,864 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c5b15f407b7629e04da 2025-12-11 20:24:06,867 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c5b15f407b7629e04da HTTP/1.1" 502 150 2025-12-11 20:24:06,867 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c5b15f407b7629e04da'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:06,882 - lava_callback - INFO - Received callback for node 693af05afee8383e92b69134 INFO: 172.21.0.1:48258 - "POST /node/693af05afee8383e92b69134 HTTP/1.0" 202 Accepted 2025-12-11 20:24:06,889 - lava_callback - INFO - Starting async job submit for node 693af05afee8383e92b69134 2025-12-11 20:24:06,913 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:24:06,939 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af05afee8383e92b69134 HTTP/1.1" 502 150 2025-12-11 20:24:06,939 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af05afee8383e92b69134'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:06,939 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af05afee8383e92b69134 2025-12-11 20:24:06,940 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af05afee8383e92b69134 HTTP/1.1" 502 150 2025-12-11 20:24:06,940 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af05afee8383e92b69134'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:07,215 - lava_callback - INFO - Received callback for node 693ae879fee8383e92b65965 INFO: 172.21.0.1:48260 - "POST /node/693ae879fee8383e92b65965 HTTP/1.0" 202 Accepted 2025-12-11 20:24:07,222 - lava_callback - INFO - Starting async job submit for node 693ae879fee8383e92b65965 2025-12-11 20:24:07,247 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:24:07,256 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae879fee8383e92b65965 HTTP/1.1" 502 150 2025-12-11 20:24:07,256 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae879fee8383e92b65965'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:07,256 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae879fee8383e92b65965 2025-12-11 20:24:07,257 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae879fee8383e92b65965 HTTP/1.1" 502 150 2025-12-11 20:24:07,257 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae879fee8383e92b65965'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:08,361 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af7b5fee8383e92b6c180 2025-12-11 20:24:08,362 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af7b5fee8383e92b6c180 HTTP/1.1" 502 150 2025-12-11 20:24:08,362 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af7b5fee8383e92b6c180'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:08,940 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af05afee8383e92b69134 2025-12-11 20:24:08,944 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af05afee8383e92b69134 HTTP/1.1" 502 150 2025-12-11 20:24:08,944 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af05afee8383e92b69134'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:09,257 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae879fee8383e92b65965 2025-12-11 20:24:09,259 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae879fee8383e92b65965 HTTP/1.1" 502 150 2025-12-11 20:24:09,259 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae879fee8383e92b65965'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:10,867 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c5b15f407b7629e04da 2025-12-11 20:24:10,868 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c5b15f407b7629e04da HTTP/1.1" 502 150 2025-12-11 20:24:10,869 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c5b15f407b7629e04da'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:11,272 - lava_callback - INFO - Received callback for node 693b083cfee8383e92b71244 INFO: 172.21.0.1:42004 - "POST /node/693b083cfee8383e92b71244 HTTP/1.0" 202 Accepted 2025-12-11 20:24:11,286 - lava_callback - INFO - Starting async job submit for node 693b083cfee8383e92b71244 2025-12-11 20:24:11,361 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:24:11,389 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b083cfee8383e92b71244 HTTP/1.1" 502 150 2025-12-11 20:24:11,389 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b083cfee8383e92b71244'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:11,389 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b083cfee8383e92b71244 2025-12-11 20:24:11,390 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b083cfee8383e92b71244 HTTP/1.1" 502 150 2025-12-11 20:24:11,390 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b083cfee8383e92b71244'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:12,363 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af7b5fee8383e92b6c180 2025-12-11 20:24:12,364 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af7b5fee8383e92b6c180 HTTP/1.1" 502 150 2025-12-11 20:24:12,364 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af7b5fee8383e92b6c180'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:12,945 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af05afee8383e92b69134 2025-12-11 20:24:12,946 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af05afee8383e92b69134 HTTP/1.1" 502 150 2025-12-11 20:24:12,946 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af05afee8383e92b69134'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:13,259 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae879fee8383e92b65965 2025-12-11 20:24:13,260 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae879fee8383e92b65965 HTTP/1.1" 502 150 2025-12-11 20:24:13,260 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae879fee8383e92b65965'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:13,390 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b083cfee8383e92b71244 2025-12-11 20:24:13,391 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b083cfee8383e92b71244 HTTP/1.1" 502 150 2025-12-11 20:24:13,392 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b083cfee8383e92b71244'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:42016 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:42032 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:24:17,392 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b083cfee8383e92b71244 2025-12-11 20:24:17,393 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b083cfee8383e92b71244 HTTP/1.1" 502 150 2025-12-11 20:24:17,393 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b083cfee8383e92b71244'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:18,869 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c5b15f407b7629e04da 2025-12-11 20:24:18,870 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c5b15f407b7629e04da HTTP/1.1" 502 150 2025-12-11 20:24:18,870 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c5b15f407b7629e04da'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:20,364 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af7b5fee8383e92b6c180 2025-12-11 20:24:20,370 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af7b5fee8383e92b6c180 HTTP/1.1" 502 150 2025-12-11 20:24:20,370 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af7b5fee8383e92b6c180'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:20,947 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af05afee8383e92b69134 2025-12-11 20:24:20,948 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af05afee8383e92b69134 HTTP/1.1" 502 150 2025-12-11 20:24:20,948 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af05afee8383e92b69134'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:21,261 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae879fee8383e92b65965 2025-12-11 20:24:21,267 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae879fee8383e92b65965 HTTP/1.1" 502 150 2025-12-11 20:24:21,267 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae879fee8383e92b65965'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:24,531 - lava_callback - INFO - Received callback for node 6938856763d980a974804865 INFO: 172.21.0.1:44090 - "POST /node/6938856763d980a974804865 HTTP/1.0" 202 Accepted 2025-12-11 20:24:24,536 - lava_callback - INFO - Starting async job submit for node 6938856763d980a974804865 2025-12-11 20:24:24,537 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:24:24,547 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938856763d980a974804865 HTTP/1.1" 502 150 2025-12-11 20:24:24,548 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938856763d980a974804865'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:24,548 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938856763d980a974804865 2025-12-11 20:24:24,548 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938856763d980a974804865 HTTP/1.1" 502 150 2025-12-11 20:24:24,548 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938856763d980a974804865'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:25,393 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b083cfee8383e92b71244 2025-12-11 20:24:25,395 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b083cfee8383e92b71244 HTTP/1.1" 502 150 2025-12-11 20:24:25,395 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b083cfee8383e92b71244'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:25,619 - lava_callback - INFO - Received callback for node 693ae9e2fee8383e92b65ebc INFO: 172.21.0.1:44096 - "POST /node/693ae9e2fee8383e92b65ebc HTTP/1.0" 202 Accepted 2025-12-11 20:24:25,669 - lava_callback - INFO - Starting async job submit for node 693ae9e2fee8383e92b65ebc 2025-12-11 20:24:25,712 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:24:25,745 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9e2fee8383e92b65ebc HTTP/1.1" 502 150 2025-12-11 20:24:25,745 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9e2fee8383e92b65ebc'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:25,745 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9e2fee8383e92b65ebc 2025-12-11 20:24:25,746 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9e2fee8383e92b65ebc HTTP/1.1" 502 150 2025-12-11 20:24:25,746 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9e2fee8383e92b65ebc'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:44098 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:24:26,549 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938856763d980a974804865 2025-12-11 20:24:26,549 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938856763d980a974804865 HTTP/1.1" 502 150 2025-12-11 20:24:26,550 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938856763d980a974804865'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:44112 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:24:27,746 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9e2fee8383e92b65ebc 2025-12-11 20:24:27,748 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9e2fee8383e92b65ebc HTTP/1.1" 502 150 2025-12-11 20:24:27,748 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9e2fee8383e92b65ebc'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:30,550 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938856763d980a974804865 2025-12-11 20:24:30,551 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938856763d980a974804865 HTTP/1.1" 502 150 2025-12-11 20:24:30,551 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938856763d980a974804865'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:31,748 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9e2fee8383e92b65ebc 2025-12-11 20:24:31,749 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9e2fee8383e92b65ebc HTTP/1.1" 502 150 2025-12-11 20:24:31,749 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9e2fee8383e92b65ebc'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:34,870 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c5b15f407b7629e04da 2025-12-11 20:24:34,871 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c5b15f407b7629e04da HTTP/1.1" 502 150 2025-12-11 20:24:34,872 - lava_callback - ERROR - Error processing callback for node 693b1c5b15f407b7629e04da: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1c5b15f407b7629e04da (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1c5b15f407b7629e04da (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1c5b15f407b7629e04da (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:24:36,371 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af7b5fee8383e92b6c180 2025-12-11 20:24:36,372 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af7b5fee8383e92b6c180 HTTP/1.1" 502 150 2025-12-11 20:24:36,372 - lava_callback - ERROR - Error processing callback for node 693af7b5fee8383e92b6c180: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af7b5fee8383e92b6c180 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af7b5fee8383e92b6c180 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af7b5fee8383e92b6c180 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:37164 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:24:36,948 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af05afee8383e92b69134 2025-12-11 20:24:36,949 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af05afee8383e92b69134 HTTP/1.1" 502 150 2025-12-11 20:24:36,949 - lava_callback - ERROR - Error processing callback for node 693af05afee8383e92b69134: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af05afee8383e92b69134 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af05afee8383e92b69134 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af05afee8383e92b69134 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:24:37,267 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae879fee8383e92b65965 2025-12-11 20:24:37,268 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae879fee8383e92b65965 HTTP/1.1" 502 150 2025-12-11 20:24:37,268 - lava_callback - ERROR - Error processing callback for node 693ae879fee8383e92b65965: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae879fee8383e92b65965 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae879fee8383e92b65965 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae879fee8383e92b65965 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:24:38,459 - lava_callback - INFO - Received callback for node 693b1cba15f407b7629e0655 INFO: 172.21.0.1:37170 - "POST /node/693b1cba15f407b7629e0655 HTTP/1.0" 202 Accepted 2025-12-11 20:24:38,465 - lava_callback - INFO - Starting async job submit for node 693b1cba15f407b7629e0655 2025-12-11 20:24:38,486 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:24:38,496 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1cba15f407b7629e0655 HTTP/1.1" 502 150 2025-12-11 20:24:38,496 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1cba15f407b7629e0655'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:38,496 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1cba15f407b7629e0655 2025-12-11 20:24:38,497 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1cba15f407b7629e0655 HTTP/1.1" 502 150 2025-12-11 20:24:38,497 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1cba15f407b7629e0655'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:38,551 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938856763d980a974804865 2025-12-11 20:24:38,552 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938856763d980a974804865 HTTP/1.1" 502 150 2025-12-11 20:24:38,552 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938856763d980a974804865'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:39,366 - lava_callback - INFO - Received callback for node 693b0246fee8383e92b6efd8 INFO: 172.21.0.1:37184 - "POST /node/693b0246fee8383e92b6efd8 HTTP/1.0" 202 Accepted 2025-12-11 20:24:39,374 - lava_callback - INFO - Starting async job submit for node 693b0246fee8383e92b6efd8 2025-12-11 20:24:39,424 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:24:39,457 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0246fee8383e92b6efd8 HTTP/1.1" 502 150 2025-12-11 20:24:39,457 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0246fee8383e92b6efd8'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:39,457 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0246fee8383e92b6efd8 2025-12-11 20:24:39,458 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0246fee8383e92b6efd8 HTTP/1.1" 502 150 2025-12-11 20:24:39,458 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0246fee8383e92b6efd8'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:39,749 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9e2fee8383e92b65ebc 2025-12-11 20:24:39,760 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9e2fee8383e92b65ebc HTTP/1.1" 502 150 2025-12-11 20:24:39,760 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9e2fee8383e92b65ebc'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:40,497 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1cba15f407b7629e0655 2025-12-11 20:24:40,498 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1cba15f407b7629e0655 HTTP/1.1" 502 150 2025-12-11 20:24:40,498 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1cba15f407b7629e0655'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:41,395 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b083cfee8383e92b71244 2025-12-11 20:24:41,397 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b083cfee8383e92b71244 HTTP/1.1" 502 150 2025-12-11 20:24:41,397 - lava_callback - ERROR - Error processing callback for node 693b083cfee8383e92b71244: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b083cfee8383e92b71244 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b083cfee8383e92b71244 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b083cfee8383e92b71244 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:24:41,458 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0246fee8383e92b6efd8 2025-12-11 20:24:41,459 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0246fee8383e92b6efd8 HTTP/1.1" 502 150 2025-12-11 20:24:41,459 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0246fee8383e92b6efd8'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:44,499 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1cba15f407b7629e0655 2025-12-11 20:24:44,500 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1cba15f407b7629e0655 HTTP/1.1" 502 150 2025-12-11 20:24:44,500 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1cba15f407b7629e0655'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:45,460 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0246fee8383e92b6efd8 2025-12-11 20:24:45,461 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0246fee8383e92b6efd8 HTTP/1.1" 502 150 2025-12-11 20:24:45,461 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0246fee8383e92b6efd8'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:46,233 - lava_callback - INFO - Received callback for node 693b0e15fee8383e92b7249d INFO: 172.21.0.1:50020 - "POST /node/693b0e15fee8383e92b7249d HTTP/1.0" 202 Accepted 2025-12-11 20:24:46,239 - lava_callback - INFO - Starting async job submit for node 693b0e15fee8383e92b7249d 2025-12-11 20:24:46,262 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:24:46,275 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e15fee8383e92b7249d HTTP/1.1" 502 150 2025-12-11 20:24:46,275 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e15fee8383e92b7249d'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:46,275 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e15fee8383e92b7249d 2025-12-11 20:24:46,275 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e15fee8383e92b7249d HTTP/1.1" 502 150 2025-12-11 20:24:46,276 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e15fee8383e92b7249d'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:50026 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:24:47,727 - lava_callback - INFO - Received callback for node 693ae87bfee8383e92b65968 INFO: 172.21.0.1:50034 - "POST /node/693ae87bfee8383e92b65968 HTTP/1.0" 202 Accepted 2025-12-11 20:24:47,738 - lava_callback - INFO - Starting async job submit for node 693ae87bfee8383e92b65968 2025-12-11 20:24:47,764 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:24:47,774 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae87bfee8383e92b65968 HTTP/1.1" 502 150 2025-12-11 20:24:47,774 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae87bfee8383e92b65968'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:47,775 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae87bfee8383e92b65968 2025-12-11 20:24:47,775 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae87bfee8383e92b65968 HTTP/1.1" 502 150 2025-12-11 20:24:47,775 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae87bfee8383e92b65968'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:48,276 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e15fee8383e92b7249d 2025-12-11 20:24:48,277 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e15fee8383e92b7249d HTTP/1.1" 502 150 2025-12-11 20:24:48,277 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e15fee8383e92b7249d'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:49,776 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae87bfee8383e92b65968 2025-12-11 20:24:49,777 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae87bfee8383e92b65968 HTTP/1.1" 502 150 2025-12-11 20:24:49,777 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae87bfee8383e92b65968'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:50,255 - lava_callback - INFO - Received callback for node 693b087bfee8383e92b7133b INFO: 172.21.0.1:60366 - "POST /node/693b087bfee8383e92b7133b HTTP/1.0" 202 Accepted 2025-12-11 20:24:50,265 - lava_callback - INFO - Starting async job submit for node 693b087bfee8383e92b7133b 2025-12-11 20:24:50,327 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:24:50,339 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b087bfee8383e92b7133b HTTP/1.1" 502 150 2025-12-11 20:24:50,339 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b087bfee8383e92b7133b'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:50,339 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b087bfee8383e92b7133b 2025-12-11 20:24:50,340 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b087bfee8383e92b7133b HTTP/1.1" 502 150 2025-12-11 20:24:50,340 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b087bfee8383e92b7133b'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:51,306 - lava_callback - INFO - Received callback for node 693b0ccffee8383e92b720aa INFO: 172.21.0.1:60378 - "POST /node/693b0ccffee8383e92b720aa HTTP/1.0" 202 Accepted 2025-12-11 20:24:51,318 - lava_callback - INFO - Starting async job submit for node 693b0ccffee8383e92b720aa 2025-12-11 20:24:51,368 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:24:51,381 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0ccffee8383e92b720aa HTTP/1.1" 502 150 2025-12-11 20:24:51,381 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0ccffee8383e92b720aa'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:51,382 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0ccffee8383e92b720aa 2025-12-11 20:24:51,383 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0ccffee8383e92b720aa HTTP/1.1" 502 150 2025-12-11 20:24:51,383 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0ccffee8383e92b720aa'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:52,277 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e15fee8383e92b7249d 2025-12-11 20:24:52,278 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e15fee8383e92b7249d HTTP/1.1" 502 150 2025-12-11 20:24:52,278 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e15fee8383e92b7249d'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:52,341 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b087bfee8383e92b7133b 2025-12-11 20:24:52,342 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b087bfee8383e92b7133b HTTP/1.1" 502 150 2025-12-11 20:24:52,342 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b087bfee8383e92b7133b'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:52,500 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1cba15f407b7629e0655 2025-12-11 20:24:52,501 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1cba15f407b7629e0655 HTTP/1.1" 502 150 2025-12-11 20:24:52,501 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1cba15f407b7629e0655'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:53,383 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0ccffee8383e92b720aa 2025-12-11 20:24:53,384 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0ccffee8383e92b720aa HTTP/1.1" 502 150 2025-12-11 20:24:53,384 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0ccffee8383e92b720aa'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:53,461 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0246fee8383e92b6efd8 2025-12-11 20:24:53,462 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0246fee8383e92b6efd8 HTTP/1.1" 502 150 2025-12-11 20:24:53,462 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0246fee8383e92b6efd8'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:53,777 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae87bfee8383e92b65968 2025-12-11 20:24:53,778 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae87bfee8383e92b65968 HTTP/1.1" 502 150 2025-12-11 20:24:53,778 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae87bfee8383e92b65968'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:24:54,552 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938856763d980a974804865 2025-12-11 20:24:54,554 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938856763d980a974804865 HTTP/1.1" 502 150 2025-12-11 20:24:54,554 - lava_callback - ERROR - Error processing callback for node 6938856763d980a974804865: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938856763d980a974804865 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938856763d980a974804865 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938856763d980a974804865 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:24:55,760 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9e2fee8383e92b65ebc 2025-12-11 20:24:55,761 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9e2fee8383e92b65ebc HTTP/1.1" 502 150 2025-12-11 20:24:55,761 - lava_callback - ERROR - Error processing callback for node 693ae9e2fee8383e92b65ebc: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9e2fee8383e92b65ebc (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9e2fee8383e92b65ebc (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9e2fee8383e92b65ebc (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:24:56,342 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b087bfee8383e92b7133b 2025-12-11 20:24:56,343 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b087bfee8383e92b7133b HTTP/1.1" 502 150 2025-12-11 20:24:56,343 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b087bfee8383e92b7133b'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:60382 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:24:57,385 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0ccffee8383e92b720aa 2025-12-11 20:24:57,386 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0ccffee8383e92b720aa HTTP/1.1" 502 150 2025-12-11 20:24:57,386 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0ccffee8383e92b720aa'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:25:00,278 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e15fee8383e92b7249d 2025-12-11 20:25:00,280 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e15fee8383e92b7249d HTTP/1.1" 502 150 2025-12-11 20:25:00,280 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e15fee8383e92b7249d'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:25:01,778 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae87bfee8383e92b65968 2025-12-11 20:25:01,780 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae87bfee8383e92b65968 HTTP/1.1" 502 150 2025-12-11 20:25:01,780 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae87bfee8383e92b65968'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:25:04,343 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b087bfee8383e92b7133b 2025-12-11 20:25:04,346 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b087bfee8383e92b7133b HTTP/1.1" 502 150 2025-12-11 20:25:04,346 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b087bfee8383e92b7133b'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:25:05,386 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0ccffee8383e92b720aa 2025-12-11 20:25:05,388 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0ccffee8383e92b720aa HTTP/1.1" 502 150 2025-12-11 20:25:05,388 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0ccffee8383e92b720aa'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:47252 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:25:08,501 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1cba15f407b7629e0655 2025-12-11 20:25:08,502 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1cba15f407b7629e0655 HTTP/1.1" 502 150 2025-12-11 20:25:08,503 - lava_callback - ERROR - Error processing callback for node 693b1cba15f407b7629e0655: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1cba15f407b7629e0655 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1cba15f407b7629e0655 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1cba15f407b7629e0655 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:25:09,462 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0246fee8383e92b6efd8 2025-12-11 20:25:09,463 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0246fee8383e92b6efd8 HTTP/1.1" 502 150 2025-12-11 20:25:09,464 - lava_callback - ERROR - Error processing callback for node 693b0246fee8383e92b6efd8: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0246fee8383e92b6efd8 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0246fee8383e92b6efd8 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0246fee8383e92b6efd8 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:25:13,098 - lava_callback - INFO - Received callback for node 693b091dfee8383e92b714ec INFO: 172.21.0.1:43172 - "POST /node/693b091dfee8383e92b714ec HTTP/1.0" 202 Accepted 2025-12-11 20:25:13,109 - lava_callback - INFO - Starting async job submit for node 693b091dfee8383e92b714ec 2025-12-11 20:25:13,173 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:25:13,208 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b091dfee8383e92b714ec HTTP/1.1" 502 150 2025-12-11 20:25:13,209 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b091dfee8383e92b714ec'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:25:13,209 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b091dfee8383e92b714ec 2025-12-11 20:25:13,210 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b091dfee8383e92b714ec HTTP/1.1" 502 150 2025-12-11 20:25:13,210 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b091dfee8383e92b714ec'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:43180 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 20:25:15,210 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b091dfee8383e92b714ec 2025-12-11 20:25:15,211 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b091dfee8383e92b714ec HTTP/1.1" 502 150 2025-12-11 20:25:15,211 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b091dfee8383e92b714ec'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:25:16,280 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e15fee8383e92b7249d 2025-12-11 20:25:16,281 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e15fee8383e92b7249d HTTP/1.1" 502 150 2025-12-11 20:25:16,281 - lava_callback - ERROR - Error processing callback for node 693b0e15fee8383e92b7249d: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e15fee8383e92b7249d (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e15fee8383e92b7249d (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e15fee8383e92b7249d (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:43184 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:25:17,780 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae87bfee8383e92b65968 2025-12-11 20:25:17,781 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae87bfee8383e92b65968 HTTP/1.1" 502 150 2025-12-11 20:25:17,781 - lava_callback - ERROR - Error processing callback for node 693ae87bfee8383e92b65968: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae87bfee8383e92b65968 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae87bfee8383e92b65968 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae87bfee8383e92b65968 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:25:19,212 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b091dfee8383e92b714ec 2025-12-11 20:25:19,219 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b091dfee8383e92b714ec HTTP/1.1" 502 150 2025-12-11 20:25:19,219 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b091dfee8383e92b714ec'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:25:20,347 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b087bfee8383e92b7133b 2025-12-11 20:25:20,348 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b087bfee8383e92b7133b HTTP/1.1" 502 150 2025-12-11 20:25:20,348 - lava_callback - ERROR - Error processing callback for node 693b087bfee8383e92b7133b: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b087bfee8383e92b7133b (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b087bfee8383e92b7133b (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b087bfee8383e92b7133b (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:25:21,389 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0ccffee8383e92b720aa 2025-12-11 20:25:21,390 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0ccffee8383e92b720aa HTTP/1.1" 502 150 2025-12-11 20:25:21,390 - lava_callback - ERROR - Error processing callback for node 693b0ccffee8383e92b720aa: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0ccffee8383e92b720aa (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0ccffee8383e92b720aa (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0ccffee8383e92b720aa (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:25:26,428 - lava_callback - INFO - Received callback for node 693ae880fee8383e92b6597a INFO: 172.21.0.1:49710 - "POST /node/693ae880fee8383e92b6597a HTTP/1.0" 202 Accepted 2025-12-11 20:25:26,435 - lava_callback - INFO - Starting async job submit for node 693ae880fee8383e92b6597a 2025-12-11 20:25:26,468 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:25:26,508 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae880fee8383e92b6597a HTTP/1.1" 502 150 2025-12-11 20:25:26,508 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae880fee8383e92b6597a'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:25:26,508 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae880fee8383e92b6597a 2025-12-11 20:25:26,509 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae880fee8383e92b6597a HTTP/1.1" 502 150 2025-12-11 20:25:26,509 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae880fee8383e92b6597a'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:49712 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:25:27,219 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b091dfee8383e92b714ec 2025-12-11 20:25:27,220 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b091dfee8383e92b714ec HTTP/1.1" 502 150 2025-12-11 20:25:27,221 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b091dfee8383e92b714ec'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:49720 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:25:28,509 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae880fee8383e92b6597a 2025-12-11 20:25:28,511 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae880fee8383e92b6597a HTTP/1.1" 502 150 2025-12-11 20:25:28,511 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae880fee8383e92b6597a'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:25:32,511 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae880fee8383e92b6597a 2025-12-11 20:25:32,512 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae880fee8383e92b6597a HTTP/1.1" 502 150 2025-12-11 20:25:32,512 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae880fee8383e92b6597a'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:47616 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:25:40,512 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae880fee8383e92b6597a 2025-12-11 20:25:40,514 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae880fee8383e92b6597a HTTP/1.1" 502 150 2025-12-11 20:25:40,514 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae880fee8383e92b6597a'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:25:43,221 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b091dfee8383e92b714ec 2025-12-11 20:25:43,222 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b091dfee8383e92b714ec HTTP/1.1" 502 150 2025-12-11 20:25:43,222 - lava_callback - ERROR - Error processing callback for node 693b091dfee8383e92b714ec: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b091dfee8383e92b714ec (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b091dfee8383e92b714ec (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b091dfee8383e92b714ec (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:51788 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:25:49,936 - lava_callback - INFO - Received callback for node 693af2d8fee8383e92b6a654 INFO: 172.21.0.1:44554 - "POST /node/693af2d8fee8383e92b6a654 HTTP/1.0" 202 Accepted 2025-12-11 20:25:49,942 - lava_callback - INFO - Starting async job submit for node 693af2d8fee8383e92b6a654 2025-12-11 20:25:49,964 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:25:49,974 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af2d8fee8383e92b6a654 HTTP/1.1" 502 150 2025-12-11 20:25:49,974 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af2d8fee8383e92b6a654'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:25:49,974 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af2d8fee8383e92b6a654 2025-12-11 20:25:49,975 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af2d8fee8383e92b6a654 HTTP/1.1" 502 150 2025-12-11 20:25:49,975 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af2d8fee8383e92b6a654'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:25:51,975 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af2d8fee8383e92b6a654 2025-12-11 20:25:51,976 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af2d8fee8383e92b6a654 HTTP/1.1" 502 150 2025-12-11 20:25:51,976 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af2d8fee8383e92b6a654'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:25:55,976 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af2d8fee8383e92b6a654 2025-12-11 20:25:55,977 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af2d8fee8383e92b6a654 HTTP/1.1" 502 150 2025-12-11 20:25:55,977 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af2d8fee8383e92b6a654'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:44562 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:25:56,514 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae880fee8383e92b6597a 2025-12-11 20:25:56,516 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae880fee8383e92b6597a HTTP/1.1" 502 150 2025-12-11 20:25:56,516 - lava_callback - ERROR - Error processing callback for node 693ae880fee8383e92b6597a: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae880fee8383e92b6597a (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae880fee8383e92b6597a (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae880fee8383e92b6597a (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:26:03,978 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af2d8fee8383e92b6a654 2025-12-11 20:26:03,979 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af2d8fee8383e92b6a654 HTTP/1.1" 502 150 2025-12-11 20:26:03,979 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af2d8fee8383e92b6a654'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:39396 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:26:07,381 - lava_callback - INFO - Received callback for node 693ae881fee8383e92b65983 INFO: 172.21.0.1:39410 - "POST /node/693ae881fee8383e92b65983 HTTP/1.0" 202 Accepted 2025-12-11 20:26:07,388 - lava_callback - INFO - Starting async job submit for node 693ae881fee8383e92b65983 2025-12-11 20:26:07,432 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:26:07,464 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae881fee8383e92b65983 HTTP/1.1" 502 150 2025-12-11 20:26:07,465 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae881fee8383e92b65983'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:26:07,465 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae881fee8383e92b65983 2025-12-11 20:26:07,465 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae881fee8383e92b65983 HTTP/1.1" 502 150 2025-12-11 20:26:07,465 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae881fee8383e92b65983'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:26:09,466 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae881fee8383e92b65983 2025-12-11 20:26:09,467 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae881fee8383e92b65983 HTTP/1.1" 502 150 2025-12-11 20:26:09,467 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae881fee8383e92b65983'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:26:12,023 - lava_callback - INFO - Received callback for node 693b126afee8383e92b73cbb INFO: 172.21.0.1:45870 - "POST /node/693b126afee8383e92b73cbb HTTP/1.0" 202 Accepted 2025-12-11 20:26:12,038 - lava_callback - INFO - Starting async job submit for node 693b126afee8383e92b73cbb 2025-12-11 20:26:12,072 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:26:12,094 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b126afee8383e92b73cbb HTTP/1.1" 502 150 2025-12-11 20:26:12,094 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b126afee8383e92b73cbb'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:26:12,094 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b126afee8383e92b73cbb 2025-12-11 20:26:12,095 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b126afee8383e92b73cbb HTTP/1.1" 502 150 2025-12-11 20:26:12,095 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b126afee8383e92b73cbb'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:26:13,467 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae881fee8383e92b65983 2025-12-11 20:26:13,469 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae881fee8383e92b65983 HTTP/1.1" 502 150 2025-12-11 20:26:13,469 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae881fee8383e92b65983'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:26:14,096 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b126afee8383e92b73cbb 2025-12-11 20:26:14,097 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b126afee8383e92b73cbb HTTP/1.1" 502 150 2025-12-11 20:26:14,097 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b126afee8383e92b73cbb'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:45886 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:45900 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:26:18,097 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b126afee8383e92b73cbb 2025-12-11 20:26:18,099 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b126afee8383e92b73cbb HTTP/1.1" 502 150 2025-12-11 20:26:18,099 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b126afee8383e92b73cbb'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:26:19,979 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af2d8fee8383e92b6a654 2025-12-11 20:26:19,980 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af2d8fee8383e92b6a654 HTTP/1.1" 502 150 2025-12-11 20:26:19,981 - lava_callback - ERROR - Error processing callback for node 693af2d8fee8383e92b6a654: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af2d8fee8383e92b6a654 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af2d8fee8383e92b6a654 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af2d8fee8383e92b6a654 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:26:21,469 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae881fee8383e92b65983 2025-12-11 20:26:21,470 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae881fee8383e92b65983 HTTP/1.1" 502 150 2025-12-11 20:26:21,470 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae881fee8383e92b65983'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:26:26,099 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b126afee8383e92b73cbb 2025-12-11 20:26:26,100 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b126afee8383e92b73cbb HTTP/1.1" 502 150 2025-12-11 20:26:26,100 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b126afee8383e92b73cbb'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:37078 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:26:27,206 - lava_callback - INFO - Received callback for node 693b0e68fee8383e92b72625 INFO: 172.21.0.1:37092 - "POST /node/693b0e68fee8383e92b72625 HTTP/1.0" 202 Accepted 2025-12-11 20:26:27,212 - lava_callback - INFO - Starting async job submit for node 693b0e68fee8383e92b72625 2025-12-11 20:26:27,235 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:26:27,246 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e68fee8383e92b72625 HTTP/1.1" 502 150 2025-12-11 20:26:27,246 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e68fee8383e92b72625'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:26:27,246 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e68fee8383e92b72625 2025-12-11 20:26:27,247 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e68fee8383e92b72625 HTTP/1.1" 502 150 2025-12-11 20:26:27,247 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e68fee8383e92b72625'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:26:27,318 - lava_callback - INFO - Received callback for node 693ae912fee8383e92b65b95 INFO: 172.21.0.1:37104 - "POST /node/693ae912fee8383e92b65b95 HTTP/1.0" 202 Accepted 2025-12-11 20:26:27,329 - lava_callback - INFO - Starting async job submit for node 693ae912fee8383e92b65b95 2025-12-11 20:26:27,363 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:26:27,398 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae912fee8383e92b65b95 HTTP/1.1" 502 150 2025-12-11 20:26:27,398 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae912fee8383e92b65b95'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:26:27,398 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae912fee8383e92b65b95 2025-12-11 20:26:27,399 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae912fee8383e92b65b95 HTTP/1.1" 502 150 2025-12-11 20:26:27,399 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae912fee8383e92b65b95'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:37118 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:26:29,247 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e68fee8383e92b72625 2025-12-11 20:26:29,249 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e68fee8383e92b72625 HTTP/1.1" 502 150 2025-12-11 20:26:29,249 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e68fee8383e92b72625'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:26:29,399 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae912fee8383e92b65b95 2025-12-11 20:26:29,400 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae912fee8383e92b65b95 HTTP/1.1" 502 150 2025-12-11 20:26:29,400 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae912fee8383e92b65b95'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:26:33,249 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e68fee8383e92b72625 2025-12-11 20:26:33,250 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e68fee8383e92b72625 HTTP/1.1" 502 150 2025-12-11 20:26:33,250 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e68fee8383e92b72625'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:26:33,401 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae912fee8383e92b65b95 2025-12-11 20:26:33,402 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae912fee8383e92b65b95 HTTP/1.1" 502 150 2025-12-11 20:26:33,402 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae912fee8383e92b65b95'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:43318 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:26:37,470 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae881fee8383e92b65983 2025-12-11 20:26:37,471 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae881fee8383e92b65983 HTTP/1.1" 502 150 2025-12-11 20:26:37,472 - lava_callback - ERROR - Error processing callback for node 693ae881fee8383e92b65983: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae881fee8383e92b65983 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae881fee8383e92b65983 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae881fee8383e92b65983 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:26:40,210 - lava_callback - INFO - Received callback for node 693b1cba15f407b7629e0659 INFO: 172.21.0.1:48792 - "POST /node/693b1cba15f407b7629e0659 HTTP/1.0" 202 Accepted 2025-12-11 20:26:40,217 - lava_callback - INFO - Starting async job submit for node 693b1cba15f407b7629e0659 2025-12-11 20:26:40,238 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:26:40,248 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1cba15f407b7629e0659 HTTP/1.1" 502 150 2025-12-11 20:26:40,248 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1cba15f407b7629e0659'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:26:40,248 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1cba15f407b7629e0659 2025-12-11 20:26:40,249 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1cba15f407b7629e0659 HTTP/1.1" 502 150 2025-12-11 20:26:40,249 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1cba15f407b7629e0659'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:26:41,250 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e68fee8383e92b72625 2025-12-11 20:26:41,252 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e68fee8383e92b72625 HTTP/1.1" 502 150 2025-12-11 20:26:41,252 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e68fee8383e92b72625'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:26:41,402 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae912fee8383e92b65b95 2025-12-11 20:26:41,403 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae912fee8383e92b65b95 HTTP/1.1" 502 150 2025-12-11 20:26:41,403 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae912fee8383e92b65b95'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:26:42,101 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b126afee8383e92b73cbb 2025-12-11 20:26:42,102 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b126afee8383e92b73cbb HTTP/1.1" 502 150 2025-12-11 20:26:42,102 - lava_callback - ERROR - Error processing callback for node 693b126afee8383e92b73cbb: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b126afee8383e92b73cbb (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b126afee8383e92b73cbb (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b126afee8383e92b73cbb (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:26:42,249 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1cba15f407b7629e0659 2025-12-11 20:26:42,250 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1cba15f407b7629e0659 HTTP/1.1" 502 150 2025-12-11 20:26:42,250 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1cba15f407b7629e0659'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:26:46,250 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1cba15f407b7629e0659 2025-12-11 20:26:46,251 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1cba15f407b7629e0659 HTTP/1.1" 502 150 2025-12-11 20:26:46,252 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1cba15f407b7629e0659'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:48804 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:26:54,252 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1cba15f407b7629e0659 2025-12-11 20:26:54,253 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1cba15f407b7629e0659 HTTP/1.1" 502 150 2025-12-11 20:26:54,253 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1cba15f407b7629e0659'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:52716 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:26:57,252 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e68fee8383e92b72625 2025-12-11 20:26:57,254 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e68fee8383e92b72625 HTTP/1.1" 502 150 2025-12-11 20:26:57,254 - lava_callback - ERROR - Error processing callback for node 693b0e68fee8383e92b72625: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e68fee8383e92b72625 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e68fee8383e92b72625 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e68fee8383e92b72625 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:26:57,403 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae912fee8383e92b65b95 2025-12-11 20:26:57,405 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae912fee8383e92b65b95 HTTP/1.1" 502 150 2025-12-11 20:26:57,405 - lava_callback - ERROR - Error processing callback for node 693ae912fee8383e92b65b95: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae912fee8383e92b65b95 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae912fee8383e92b65b95 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae912fee8383e92b65b95 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:27:06,392 - lava_callback - INFO - Received callback for node 693af813fee8383e92b6c382 INFO: 172.21.0.1:45228 - "POST /node/693af813fee8383e92b6c382 HTTP/1.0" 202 Accepted 2025-12-11 20:27:06,398 - lava_callback - INFO - Starting async job submit for node 693af813fee8383e92b6c382 2025-12-11 20:27:06,417 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:27:06,435 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af813fee8383e92b6c382 HTTP/1.1" 502 150 2025-12-11 20:27:06,435 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af813fee8383e92b6c382'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:27:06,435 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af813fee8383e92b6c382 2025-12-11 20:27:06,435 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af813fee8383e92b6c382 HTTP/1.1" 502 150 2025-12-11 20:27:06,436 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af813fee8383e92b6c382'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:45240 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:27:07,466 - lava_callback - INFO - Received callback for node 693ae934fee8383e92b65c1b INFO: 172.21.0.1:45250 - "POST /node/693ae934fee8383e92b65c1b HTTP/1.0" 202 Accepted 2025-12-11 20:27:07,472 - lava_callback - INFO - Starting async job submit for node 693ae934fee8383e92b65c1b 2025-12-11 20:27:07,495 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:27:07,505 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae934fee8383e92b65c1b HTTP/1.1" 502 150 2025-12-11 20:27:07,505 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae934fee8383e92b65c1b'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:27:07,505 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae934fee8383e92b65c1b 2025-12-11 20:27:07,505 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae934fee8383e92b65c1b HTTP/1.1" 502 150 2025-12-11 20:27:07,505 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae934fee8383e92b65c1b'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:27:08,436 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af813fee8383e92b6c382 2025-12-11 20:27:08,437 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af813fee8383e92b6c382 HTTP/1.1" 502 150 2025-12-11 20:27:08,437 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af813fee8383e92b6c382'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:27:09,506 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae934fee8383e92b65c1b 2025-12-11 20:27:09,507 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae934fee8383e92b65c1b HTTP/1.1" 502 150 2025-12-11 20:27:09,507 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae934fee8383e92b65c1b'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:27:10,253 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1cba15f407b7629e0659 2025-12-11 20:27:10,254 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1cba15f407b7629e0659 HTTP/1.1" 502 150 2025-12-11 20:27:10,254 - lava_callback - ERROR - Error processing callback for node 693b1cba15f407b7629e0659: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1cba15f407b7629e0659 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1cba15f407b7629e0659 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1cba15f407b7629e0659 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:27:12,437 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af813fee8383e92b6c382 2025-12-11 20:27:12,438 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af813fee8383e92b6c382 HTTP/1.1" 502 150 2025-12-11 20:27:12,438 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af813fee8383e92b6c382'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:27:13,507 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae934fee8383e92b65c1b 2025-12-11 20:27:13,508 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae934fee8383e92b65c1b HTTP/1.1" 502 150 2025-12-11 20:27:13,509 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae934fee8383e92b65c1b'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:39296 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:39298 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:27:20,438 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af813fee8383e92b6c382 2025-12-11 20:27:20,440 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af813fee8383e92b6c382 HTTP/1.1" 502 150 2025-12-11 20:27:20,440 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af813fee8383e92b6c382'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:27:21,509 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae934fee8383e92b65c1b 2025-12-11 20:27:21,511 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae934fee8383e92b65c1b HTTP/1.1" 502 150 2025-12-11 20:27:21,511 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae934fee8383e92b65c1b'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:45208 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:27:26,666 - lava_callback - INFO - Received callback for node 693ae646fee8383e92b64c83 INFO: 172.21.0.1:45224 - "POST /node/693ae646fee8383e92b64c83 HTTP/1.0" 202 Accepted 2025-12-11 20:27:26,673 - lava_callback - INFO - Starting async job submit for node 693ae646fee8383e92b64c83 2025-12-11 20:27:26,752 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:27:26,772 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae646fee8383e92b64c83 HTTP/1.1" 502 150 2025-12-11 20:27:26,773 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae646fee8383e92b64c83'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:27:26,773 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae646fee8383e92b64c83 2025-12-11 20:27:26,774 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae646fee8383e92b64c83 HTTP/1.1" 502 150 2025-12-11 20:27:26,774 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae646fee8383e92b64c83'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:45240 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:27:28,228 - lava_callback - INFO - Received callback for node 693b149ffee8383e92b74468 INFO: 172.21.0.1:45242 - "POST /node/693b149ffee8383e92b74468 HTTP/1.0" 202 Accepted 2025-12-11 20:27:28,243 - lava_callback - INFO - Starting async job submit for node 693b149ffee8383e92b74468 2025-12-11 20:27:28,347 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:27:28,364 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b149ffee8383e92b74468 HTTP/1.1" 502 150 2025-12-11 20:27:28,365 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b149ffee8383e92b74468'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:27:28,365 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b149ffee8383e92b74468 2025-12-11 20:27:28,365 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b149ffee8383e92b74468 HTTP/1.1" 502 150 2025-12-11 20:27:28,365 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b149ffee8383e92b74468'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:27:28,774 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae646fee8383e92b64c83 2025-12-11 20:27:28,776 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae646fee8383e92b64c83 HTTP/1.1" 502 150 2025-12-11 20:27:28,776 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae646fee8383e92b64c83'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:27:29,023 - lava_callback - INFO - Received callback for node 693af6affee8383e92b6bca7 INFO: 172.21.0.1:45246 - "POST /node/693af6affee8383e92b6bca7 HTTP/1.0" 202 Accepted 2025-12-11 20:27:29,034 - lava_callback - INFO - Starting async job submit for node 693af6affee8383e92b6bca7 2025-12-11 20:27:29,072 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:27:29,084 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af6affee8383e92b6bca7 HTTP/1.1" 502 150 2025-12-11 20:27:29,085 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af6affee8383e92b6bca7'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:27:29,085 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af6affee8383e92b6bca7 2025-12-11 20:27:29,086 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af6affee8383e92b6bca7 HTTP/1.1" 502 150 2025-12-11 20:27:29,086 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af6affee8383e92b6bca7'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:27:29,533 - lava_callback - INFO - Received callback for node 693b097afee8383e92b71783 INFO: 172.21.0.1:45248 - "POST /node/693b097afee8383e92b71783 HTTP/1.0" 202 Accepted 2025-12-11 20:27:29,572 - lava_callback - INFO - Starting async job submit for node 693b097afee8383e92b71783 2025-12-11 20:27:29,683 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:27:29,696 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b097afee8383e92b71783 HTTP/1.1" 502 150 2025-12-11 20:27:29,696 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b097afee8383e92b71783'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:27:29,696 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b097afee8383e92b71783 2025-12-11 20:27:29,702 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b097afee8383e92b71783 HTTP/1.1" 502 150 2025-12-11 20:27:29,702 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b097afee8383e92b71783'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:27:30,366 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b149ffee8383e92b74468 2025-12-11 20:27:30,367 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b149ffee8383e92b74468 HTTP/1.1" 502 150 2025-12-11 20:27:30,367 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b149ffee8383e92b74468'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:27:31,087 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af6affee8383e92b6bca7 2025-12-11 20:27:31,088 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af6affee8383e92b6bca7 HTTP/1.1" 502 150 2025-12-11 20:27:31,088 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af6affee8383e92b6bca7'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:27:31,703 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b097afee8383e92b71783 2025-12-11 20:27:31,704 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b097afee8383e92b71783 HTTP/1.1" 502 150 2025-12-11 20:27:31,704 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b097afee8383e92b71783'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:27:32,776 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae646fee8383e92b64c83 2025-12-11 20:27:32,777 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae646fee8383e92b64c83 HTTP/1.1" 502 150 2025-12-11 20:27:32,777 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae646fee8383e92b64c83'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:27:34,367 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b149ffee8383e92b74468 2025-12-11 20:27:34,368 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b149ffee8383e92b74468 HTTP/1.1" 502 150 2025-12-11 20:27:34,368 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b149ffee8383e92b74468'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:27:35,088 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af6affee8383e92b6bca7 2025-12-11 20:27:35,090 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af6affee8383e92b6bca7 HTTP/1.1" 502 150 2025-12-11 20:27:35,090 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af6affee8383e92b6bca7'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:27:35,704 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b097afee8383e92b71783 2025-12-11 20:27:35,706 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b097afee8383e92b71783 HTTP/1.1" 502 150 2025-12-11 20:27:35,706 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b097afee8383e92b71783'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:27:36,440 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af813fee8383e92b6c382 2025-12-11 20:27:36,441 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af813fee8383e92b6c382 HTTP/1.1" 502 150 2025-12-11 20:27:36,441 - lava_callback - ERROR - Error processing callback for node 693af813fee8383e92b6c382: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af813fee8383e92b6c382 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af813fee8383e92b6c382 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af813fee8383e92b6c382 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:36172 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:27:37,512 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae934fee8383e92b65c1b 2025-12-11 20:27:37,513 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae934fee8383e92b65c1b HTTP/1.1" 502 150 2025-12-11 20:27:37,513 - lava_callback - ERROR - Error processing callback for node 693ae934fee8383e92b65c1b: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae934fee8383e92b65c1b (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae934fee8383e92b65c1b (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae934fee8383e92b65c1b (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:27:40,777 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae646fee8383e92b64c83 2025-12-11 20:27:40,778 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae646fee8383e92b64c83 HTTP/1.1" 502 150 2025-12-11 20:27:40,779 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae646fee8383e92b64c83'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:27:42,368 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b149ffee8383e92b74468 2025-12-11 20:27:42,369 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b149ffee8383e92b74468 HTTP/1.1" 502 150 2025-12-11 20:27:42,370 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b149ffee8383e92b74468'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:27:43,090 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af6affee8383e92b6bca7 2025-12-11 20:27:43,091 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af6affee8383e92b6bca7 HTTP/1.1" 502 150 2025-12-11 20:27:43,091 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af6affee8383e92b6bca7'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:27:43,706 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b097afee8383e92b71783 2025-12-11 20:27:43,707 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b097afee8383e92b71783 HTTP/1.1" 502 150 2025-12-11 20:27:43,707 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b097afee8383e92b71783'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:39556 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58278 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:27:56,779 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae646fee8383e92b64c83 2025-12-11 20:27:56,780 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae646fee8383e92b64c83 HTTP/1.1" 502 150 2025-12-11 20:27:56,780 - lava_callback - ERROR - Error processing callback for node 693ae646fee8383e92b64c83: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae646fee8383e92b64c83 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae646fee8383e92b64c83 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae646fee8383e92b64c83 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:27:58,370 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b149ffee8383e92b74468 2025-12-11 20:27:58,371 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b149ffee8383e92b74468 HTTP/1.1" 502 150 2025-12-11 20:27:58,371 - lava_callback - ERROR - Error processing callback for node 693b149ffee8383e92b74468: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b149ffee8383e92b74468 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b149ffee8383e92b74468 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b149ffee8383e92b74468 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:27:59,091 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af6affee8383e92b6bca7 2025-12-11 20:27:59,093 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af6affee8383e92b6bca7 HTTP/1.1" 502 150 2025-12-11 20:27:59,093 - lava_callback - ERROR - Error processing callback for node 693af6affee8383e92b6bca7: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af6affee8383e92b6bca7 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af6affee8383e92b6bca7 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af6affee8383e92b6bca7 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:27:59,707 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b097afee8383e92b71783 2025-12-11 20:27:59,709 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b097afee8383e92b71783 HTTP/1.1" 502 150 2025-12-11 20:27:59,709 - lava_callback - ERROR - Error processing callback for node 693b097afee8383e92b71783: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b097afee8383e92b71783 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b097afee8383e92b71783 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b097afee8383e92b71783 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:55342 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:28:06,648 - lava_callback - INFO - Received callback for node 693b0eb2fee8383e92b727d8 INFO: 172.21.0.1:55348 - "POST /node/693b0eb2fee8383e92b727d8 HTTP/1.0" 202 Accepted 2025-12-11 20:28:06,654 - lava_callback - INFO - Starting async job submit for node 693b0eb2fee8383e92b727d8 2025-12-11 20:28:06,676 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:28:06,710 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0eb2fee8383e92b727d8 HTTP/1.1" 502 150 2025-12-11 20:28:06,710 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0eb2fee8383e92b727d8'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:28:06,710 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0eb2fee8383e92b727d8 2025-12-11 20:28:06,711 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0eb2fee8383e92b727d8 HTTP/1.1" 502 150 2025-12-11 20:28:06,711 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0eb2fee8383e92b727d8'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:28:08,114 - lava_callback - INFO - Received callback for node 693b1ad715f407b7629dff56 INFO: 172.21.0.1:55354 - "POST /node/693b1ad715f407b7629dff56 HTTP/1.0" 202 Accepted 2025-12-11 20:28:08,120 - lava_callback - INFO - Starting async job submit for node 693b1ad715f407b7629dff56 2025-12-11 20:28:08,156 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:28:08,165 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ad715f407b7629dff56 HTTP/1.1" 502 150 2025-12-11 20:28:08,166 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ad715f407b7629dff56'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:28:08,166 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ad715f407b7629dff56 2025-12-11 20:28:08,166 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ad715f407b7629dff56 HTTP/1.1" 502 150 2025-12-11 20:28:08,166 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ad715f407b7629dff56'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:28:08,711 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0eb2fee8383e92b727d8 2025-12-11 20:28:08,714 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0eb2fee8383e92b727d8 HTTP/1.1" 502 150 2025-12-11 20:28:08,714 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0eb2fee8383e92b727d8'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:28:10,167 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ad715f407b7629dff56 2025-12-11 20:28:10,168 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ad715f407b7629dff56 HTTP/1.1" 502 150 2025-12-11 20:28:10,168 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ad715f407b7629dff56'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:28:12,714 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0eb2fee8383e92b727d8 2025-12-11 20:28:12,716 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0eb2fee8383e92b727d8 HTTP/1.1" 502 150 2025-12-11 20:28:12,716 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0eb2fee8383e92b727d8'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:28:14,168 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ad715f407b7629dff56 2025-12-11 20:28:14,170 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ad715f407b7629dff56 HTTP/1.1" 502 150 2025-12-11 20:28:14,170 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ad715f407b7629dff56'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:34312 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:34314 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:28:20,716 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0eb2fee8383e92b727d8 2025-12-11 20:28:20,717 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0eb2fee8383e92b727d8 HTTP/1.1" 502 150 2025-12-11 20:28:20,718 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0eb2fee8383e92b727d8'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:28:22,173 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ad715f407b7629dff56 2025-12-11 20:28:22,174 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ad715f407b7629dff56 HTTP/1.1" 502 150 2025-12-11 20:28:22,174 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ad715f407b7629dff56'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:50522 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50534 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52980 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:28:36,718 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0eb2fee8383e92b727d8 2025-12-11 20:28:36,719 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0eb2fee8383e92b727d8 HTTP/1.1" 502 150 2025-12-11 20:28:36,719 - lava_callback - ERROR - Error processing callback for node 693b0eb2fee8383e92b727d8: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0eb2fee8383e92b727d8 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0eb2fee8383e92b727d8 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0eb2fee8383e92b727d8 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:28:38,174 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ad715f407b7629dff56 2025-12-11 20:28:38,175 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ad715f407b7629dff56 HTTP/1.1" 502 150 2025-12-11 20:28:38,175 - lava_callback - ERROR - Error processing callback for node 693b1ad715f407b7629dff56: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1ad715f407b7629dff56 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1ad715f407b7629dff56 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1ad715f407b7629dff56 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:48134 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:28:47,429 - lava_callback - INFO - Received callback for node 693b0262fee8383e92b6f015 INFO: 172.21.0.1:48138 - "POST /node/693b0262fee8383e92b6f015 HTTP/1.0" 202 Accepted 2025-12-11 20:28:47,436 - lava_callback - INFO - Starting async job submit for node 693b0262fee8383e92b6f015 2025-12-11 20:28:47,489 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:28:47,512 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0262fee8383e92b6f015 HTTP/1.1" 502 150 2025-12-11 20:28:47,512 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0262fee8383e92b6f015'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:28:47,512 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0262fee8383e92b6f015 2025-12-11 20:28:47,513 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0262fee8383e92b6f015 HTTP/1.1" 502 150 2025-12-11 20:28:47,513 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0262fee8383e92b6f015'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:28:49,513 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0262fee8383e92b6f015 2025-12-11 20:28:49,514 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0262fee8383e92b6f015 HTTP/1.1" 502 150 2025-12-11 20:28:49,514 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0262fee8383e92b6f015'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:28:53,515 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0262fee8383e92b6f015 2025-12-11 20:28:53,518 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0262fee8383e92b6f015 HTTP/1.1" 502 150 2025-12-11 20:28:53,518 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0262fee8383e92b6f015'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:28:56,150 - lava_callback - INFO - Received callback for node 693af6f5fee8383e92b6be11 INFO: 172.21.0.1:59176 - "POST /node/693af6f5fee8383e92b6be11 HTTP/1.0" 202 Accepted 2025-12-11 20:28:56,160 - lava_callback - INFO - Starting async job submit for node 693af6f5fee8383e92b6be11 2025-12-11 20:28:56,207 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:28:56,237 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af6f5fee8383e92b6be11 HTTP/1.1" 502 150 2025-12-11 20:28:56,237 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af6f5fee8383e92b6be11'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:28:56,237 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af6f5fee8383e92b6be11 2025-12-11 20:28:56,238 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af6f5fee8383e92b6be11 HTTP/1.1" 502 150 2025-12-11 20:28:56,238 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af6f5fee8383e92b6be11'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:59182 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:28:58,238 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af6f5fee8383e92b6be11 2025-12-11 20:28:58,239 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af6f5fee8383e92b6be11 HTTP/1.1" 502 150 2025-12-11 20:28:58,239 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af6f5fee8383e92b6be11'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:29:01,518 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0262fee8383e92b6f015 2025-12-11 20:29:01,519 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0262fee8383e92b6f015 HTTP/1.1" 502 150 2025-12-11 20:29:01,519 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0262fee8383e92b6f015'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:29:02,240 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af6f5fee8383e92b6be11 2025-12-11 20:29:02,241 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af6f5fee8383e92b6be11 HTTP/1.1" 502 150 2025-12-11 20:29:02,241 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af6f5fee8383e92b6be11'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:29:06,498 - lava_callback - INFO - Received callback for node 693af847fee8383e92b6c4ff INFO: 172.21.0.1:53508 - "POST /node/693af847fee8383e92b6c4ff HTTP/1.0" 202 Accepted 2025-12-11 20:29:06,508 - lava_callback - INFO - Starting async job submit for node 693af847fee8383e92b6c4ff 2025-12-11 20:29:06,536 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 INFO: 172.21.0.1:53516 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:29:06,552 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af847fee8383e92b6c4ff HTTP/1.1" 502 150 2025-12-11 20:29:06,552 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af847fee8383e92b6c4ff'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:29:06,553 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af847fee8383e92b6c4ff 2025-12-11 20:29:06,553 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af847fee8383e92b6c4ff HTTP/1.1" 502 150 2025-12-11 20:29:06,553 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af847fee8383e92b6c4ff'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:29:08,554 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af847fee8383e92b6c4ff 2025-12-11 20:29:08,555 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af847fee8383e92b6c4ff HTTP/1.1" 502 150 2025-12-11 20:29:08,555 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af847fee8383e92b6c4ff'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:29:10,241 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af6f5fee8383e92b6be11 2025-12-11 20:29:10,242 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af6f5fee8383e92b6be11 HTTP/1.1" 502 150 2025-12-11 20:29:10,242 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af6f5fee8383e92b6be11'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:29:12,555 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af847fee8383e92b6c4ff 2025-12-11 20:29:12,556 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af847fee8383e92b6c4ff HTTP/1.1" 502 150 2025-12-11 20:29:12,556 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af847fee8383e92b6c4ff'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:45378 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:45390 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:29:17,519 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0262fee8383e92b6f015 2025-12-11 20:29:17,520 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0262fee8383e92b6f015 HTTP/1.1" 502 150 2025-12-11 20:29:17,521 - lava_callback - ERROR - Error processing callback for node 693b0262fee8383e92b6f015: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0262fee8383e92b6f015 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0262fee8383e92b6f015 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0262fee8383e92b6f015 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:29:20,557 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af847fee8383e92b6c4ff 2025-12-11 20:29:20,558 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af847fee8383e92b6c4ff HTTP/1.1" 502 150 2025-12-11 20:29:20,558 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af847fee8383e92b6c4ff'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:29:26,242 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af6f5fee8383e92b6be11 2025-12-11 20:29:26,244 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af6f5fee8383e92b6be11 HTTP/1.1" 502 150 2025-12-11 20:29:26,244 - lava_callback - ERROR - Error processing callback for node 693af6f5fee8383e92b6be11: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af6f5fee8383e92b6be11 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af6f5fee8383e92b6be11 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af6f5fee8383e92b6be11 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:57330 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57342 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60790 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:29:36,559 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af847fee8383e92b6c4ff 2025-12-11 20:29:36,560 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af847fee8383e92b6c4ff HTTP/1.1" 502 150 2025-12-11 20:29:36,560 - lava_callback - ERROR - Error processing callback for node 693af847fee8383e92b6c4ff: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af847fee8383e92b6c4ff (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af847fee8383e92b6c4ff (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af847fee8383e92b6c4ff (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:29:46,463 - lava_callback - INFO - Received callback for node 693b0eebfee8383e92b72939 INFO: 172.21.0.1:40886 - "POST /node/693b0eebfee8383e92b72939 HTTP/1.0" 202 Accepted 2025-12-11 20:29:46,469 - lava_callback - INFO - Starting async job submit for node 693b0eebfee8383e92b72939 2025-12-11 20:29:46,491 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:29:46,522 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0eebfee8383e92b72939 HTTP/1.1" 502 150 2025-12-11 20:29:46,522 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0eebfee8383e92b72939'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:29:46,523 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0eebfee8383e92b72939 2025-12-11 20:29:46,523 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0eebfee8383e92b72939 HTTP/1.1" 502 150 2025-12-11 20:29:46,523 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0eebfee8383e92b72939'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:40900 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:29:48,524 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0eebfee8383e92b72939 2025-12-11 20:29:48,525 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0eebfee8383e92b72939 HTTP/1.1" 502 150 2025-12-11 20:29:48,525 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0eebfee8383e92b72939'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:29:52,525 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0eebfee8383e92b72939 2025-12-11 20:29:52,526 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0eebfee8383e92b72939 HTTP/1.1" 502 150 2025-12-11 20:29:52,526 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0eebfee8383e92b72939'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:29:56,280 - lava_callback - INFO - Received callback for node 693b181c15f407b7629df3f8 INFO: 172.21.0.1:57912 - "POST /node/693b181c15f407b7629df3f8 HTTP/1.0" 202 Accepted 2025-12-11 20:29:56,288 - lava_callback - INFO - Starting async job submit for node 693b181c15f407b7629df3f8 2025-12-11 20:29:56,307 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:29:56,317 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b181c15f407b7629df3f8 HTTP/1.1" 502 150 2025-12-11 20:29:56,317 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b181c15f407b7629df3f8'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:29:56,317 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b181c15f407b7629df3f8 2025-12-11 20:29:56,318 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b181c15f407b7629df3f8 HTTP/1.1" 502 150 2025-12-11 20:29:56,318 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b181c15f407b7629df3f8'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:57914 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:29:58,318 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b181c15f407b7629df3f8 2025-12-11 20:29:58,319 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b181c15f407b7629df3f8 HTTP/1.1" 502 150 2025-12-11 20:29:58,319 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b181c15f407b7629df3f8'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:30:00,526 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0eebfee8383e92b72939 2025-12-11 20:30:00,527 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0eebfee8383e92b72939 HTTP/1.1" 502 150 2025-12-11 20:30:00,528 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0eebfee8383e92b72939'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:30:02,320 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b181c15f407b7629df3f8 2025-12-11 20:30:02,321 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b181c15f407b7629df3f8 HTTP/1.1" 502 150 2025-12-11 20:30:02,321 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b181c15f407b7629df3f8'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:51972 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:30:06,799 - lava_callback - INFO - Received callback for node 693b197d15f407b7629df652 INFO: 172.21.0.1:51984 - "POST /node/693b197d15f407b7629df652 HTTP/1.0" 202 Accepted 2025-12-11 20:30:06,811 - lava_callback - INFO - Starting async job submit for node 693b197d15f407b7629df652 2025-12-11 20:30:06,897 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:30:06,933 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b197d15f407b7629df652 HTTP/1.1" 502 150 2025-12-11 20:30:06,933 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b197d15f407b7629df652'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:30:06,933 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b197d15f407b7629df652 2025-12-11 20:30:06,934 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b197d15f407b7629df652 HTTP/1.1" 502 150 2025-12-11 20:30:06,934 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b197d15f407b7629df652'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:30:08,934 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b197d15f407b7629df652 2025-12-11 20:30:08,935 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b197d15f407b7629df652 HTTP/1.1" 502 150 2025-12-11 20:30:08,935 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b197d15f407b7629df652'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:30:10,321 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b181c15f407b7629df3f8 2025-12-11 20:30:10,323 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b181c15f407b7629df3f8 HTTP/1.1" 502 150 2025-12-11 20:30:10,323 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b181c15f407b7629df3f8'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:30:12,935 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b197d15f407b7629df652 2025-12-11 20:30:12,937 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b197d15f407b7629df652 HTTP/1.1" 502 150 2025-12-11 20:30:12,937 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b197d15f407b7629df652'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:44838 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:44852 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:30:16,528 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0eebfee8383e92b72939 2025-12-11 20:30:16,529 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0eebfee8383e92b72939 HTTP/1.1" 502 150 2025-12-11 20:30:16,529 - lava_callback - ERROR - Error processing callback for node 693b0eebfee8383e92b72939: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0eebfee8383e92b72939 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0eebfee8383e92b72939 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0eebfee8383e92b72939 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:30:20,937 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b197d15f407b7629df652 2025-12-11 20:30:20,938 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b197d15f407b7629df652 HTTP/1.1" 502 150 2025-12-11 20:30:20,938 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b197d15f407b7629df652'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:30:26,323 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b181c15f407b7629df3f8 2025-12-11 20:30:26,324 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b181c15f407b7629df3f8 HTTP/1.1" 502 150 2025-12-11 20:30:26,324 - lava_callback - ERROR - Error processing callback for node 693b181c15f407b7629df3f8: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b181c15f407b7629df3f8 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b181c15f407b7629df3f8 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b181c15f407b7629df3f8 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:49162 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49170 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36868 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:30:36,938 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b197d15f407b7629df652 2025-12-11 20:30:36,939 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b197d15f407b7629df652 HTTP/1.1" 502 150 2025-12-11 20:30:36,940 - lava_callback - ERROR - Error processing callback for node 693b197d15f407b7629df652: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b197d15f407b7629df652 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b197d15f407b7629df652 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b197d15f407b7629df652 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:37048 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:30:46,736 - lava_callback - INFO - Received callback for node 693ae937fee8383e92b65c2c INFO: 172.21.0.1:37058 - "POST /node/693ae937fee8383e92b65c2c HTTP/1.0" 202 Accepted 2025-12-11 20:30:46,742 - lava_callback - INFO - Starting async job submit for node 693ae937fee8383e92b65c2c 2025-12-11 20:30:46,774 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:30:46,790 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae937fee8383e92b65c2c HTTP/1.1" 502 150 2025-12-11 20:30:46,790 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae937fee8383e92b65c2c'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:30:46,790 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae937fee8383e92b65c2c 2025-12-11 20:30:46,791 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae937fee8383e92b65c2c HTTP/1.1" 502 150 2025-12-11 20:30:46,791 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae937fee8383e92b65c2c'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:30:47,957 - lava_callback - INFO - Received callback for node 693894e963d980a9748051b9 INFO: 172.21.0.1:37068 - "POST /node/693894e963d980a9748051b9 HTTP/1.0" 202 Accepted 2025-12-11 20:30:47,963 - lava_callback - INFO - Starting async job submit for node 693894e963d980a9748051b9 2025-12-11 20:30:47,994 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:30:48,004 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693894e963d980a9748051b9 HTTP/1.1" 502 150 2025-12-11 20:30:48,004 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693894e963d980a9748051b9'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:30:48,004 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693894e963d980a9748051b9 2025-12-11 20:30:48,005 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693894e963d980a9748051b9 HTTP/1.1" 502 150 2025-12-11 20:30:48,005 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693894e963d980a9748051b9'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:30:48,791 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae937fee8383e92b65c2c 2025-12-11 20:30:48,792 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae937fee8383e92b65c2c HTTP/1.1" 502 150 2025-12-11 20:30:48,792 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae937fee8383e92b65c2c'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:30:50,005 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693894e963d980a9748051b9 2025-12-11 20:30:50,006 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693894e963d980a9748051b9 HTTP/1.1" 502 150 2025-12-11 20:30:50,006 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693894e963d980a9748051b9'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:30:52,792 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae937fee8383e92b65c2c 2025-12-11 20:30:52,793 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae937fee8383e92b65c2c HTTP/1.1" 502 150 2025-12-11 20:30:52,794 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae937fee8383e92b65c2c'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:30:54,007 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693894e963d980a9748051b9 2025-12-11 20:30:54,009 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693894e963d980a9748051b9 HTTP/1.1" 502 150 2025-12-11 20:30:54,009 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693894e963d980a9748051b9'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:50262 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:31:00,794 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae937fee8383e92b65c2c 2025-12-11 20:31:00,796 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae937fee8383e92b65c2c HTTP/1.1" 502 150 2025-12-11 20:31:00,796 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae937fee8383e92b65c2c'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:31:02,010 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693894e963d980a9748051b9 2025-12-11 20:31:02,011 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693894e963d980a9748051b9 HTTP/1.1" 502 150 2025-12-11 20:31:02,011 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693894e963d980a9748051b9'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:32786 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:31:07,595 - lava_callback - INFO - Received callback for node 693ae9cffee8383e92b65e67 INFO: 172.21.0.1:32798 - "POST /node/693ae9cffee8383e92b65e67 HTTP/1.0" 202 Accepted 2025-12-11 20:31:07,602 - lava_callback - INFO - Starting async job submit for node 693ae9cffee8383e92b65e67 2025-12-11 20:31:07,627 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:31:07,637 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9cffee8383e92b65e67 HTTP/1.1" 502 150 2025-12-11 20:31:07,637 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9cffee8383e92b65e67'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:31:07,637 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9cffee8383e92b65e67 2025-12-11 20:31:07,638 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9cffee8383e92b65e67 HTTP/1.1" 502 150 2025-12-11 20:31:07,638 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9cffee8383e92b65e67'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:31:09,638 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9cffee8383e92b65e67 2025-12-11 20:31:09,640 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9cffee8383e92b65e67 HTTP/1.1" 502 150 2025-12-11 20:31:09,640 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9cffee8383e92b65e67'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:31:13,640 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9cffee8383e92b65e67 2025-12-11 20:31:13,641 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9cffee8383e92b65e67 HTTP/1.1" 502 150 2025-12-11 20:31:13,641 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9cffee8383e92b65e67'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:56122 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:56128 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:31:16,796 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae937fee8383e92b65c2c 2025-12-11 20:31:16,797 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae937fee8383e92b65c2c HTTP/1.1" 502 150 2025-12-11 20:31:16,797 - lava_callback - ERROR - Error processing callback for node 693ae937fee8383e92b65c2c: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae937fee8383e92b65c2c (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae937fee8383e92b65c2c (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae937fee8383e92b65c2c (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:31:18,011 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693894e963d980a9748051b9 2025-12-11 20:31:18,013 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693894e963d980a9748051b9 HTTP/1.1" 502 150 2025-12-11 20:31:18,013 - lava_callback - ERROR - Error processing callback for node 693894e963d980a9748051b9: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693894e963d980a9748051b9 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693894e963d980a9748051b9 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693894e963d980a9748051b9 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:31:21,642 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9cffee8383e92b65e67 2025-12-11 20:31:21,643 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9cffee8383e92b65e67 HTTP/1.1" 502 150 2025-12-11 20:31:21,643 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9cffee8383e92b65e67'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:59478 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59480 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40890 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:31:37,643 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9cffee8383e92b65e67 2025-12-11 20:31:37,644 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9cffee8383e92b65e67 HTTP/1.1" 502 150 2025-12-11 20:31:37,645 - lava_callback - ERROR - Error processing callback for node 693ae9cffee8383e92b65e67: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9cffee8383e92b65e67 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9cffee8383e92b65e67 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9cffee8383e92b65e67 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:42080 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:31:48,048 - lava_callback - INFO - Received callback for node 693ae9d5fee8383e92b65e8b INFO: 172.21.0.1:42096 - "POST /node/693ae9d5fee8383e92b65e8b HTTP/1.0" 202 Accepted 2025-12-11 20:31:48,064 - lava_callback - INFO - Starting async job submit for node 693ae9d5fee8383e92b65e8b 2025-12-11 20:31:48,533 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:31:48,556 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9d5fee8383e92b65e8b HTTP/1.1" 502 150 2025-12-11 20:31:48,556 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9d5fee8383e92b65e8b'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:31:48,556 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9d5fee8383e92b65e8b 2025-12-11 20:31:48,557 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9d5fee8383e92b65e8b HTTP/1.1" 502 150 2025-12-11 20:31:48,557 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9d5fee8383e92b65e8b'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:31:50,557 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9d5fee8383e92b65e8b 2025-12-11 20:31:50,558 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9d5fee8383e92b65e8b HTTP/1.1" 502 150 2025-12-11 20:31:50,558 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9d5fee8383e92b65e8b'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:31:54,558 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9d5fee8383e92b65e8b 2025-12-11 20:31:54,560 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9d5fee8383e92b65e8b HTTP/1.1" 502 150 2025-12-11 20:31:54,560 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9d5fee8383e92b65e8b'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:40026 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:32:02,560 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9d5fee8383e92b65e8b 2025-12-11 20:32:02,561 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9d5fee8383e92b65e8b HTTP/1.1" 502 150 2025-12-11 20:32:02,561 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9d5fee8383e92b65e8b'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:46892 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:32:06,571 - lava_callback - INFO - Received callback for node 693afabefee8383e92b6d29b INFO: 172.21.0.1:46896 - "POST /node/693afabefee8383e92b6d29b HTTP/1.0" 202 Accepted 2025-12-11 20:32:06,577 - lava_callback - INFO - Starting async job submit for node 693afabefee8383e92b6d29b 2025-12-11 20:32:06,595 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:32:06,605 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afabefee8383e92b6d29b HTTP/1.1" 502 150 2025-12-11 20:32:06,605 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afabefee8383e92b6d29b'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:32:06,605 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afabefee8383e92b6d29b 2025-12-11 20:32:06,605 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afabefee8383e92b6d29b HTTP/1.1" 502 150 2025-12-11 20:32:06,606 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afabefee8383e92b6d29b'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:32:07,722 - lava_callback - INFO - Received callback for node 693b0d14fee8383e92b72156 INFO: 172.21.0.1:46906 - "POST /node/693b0d14fee8383e92b72156 HTTP/1.0" 202 Accepted 2025-12-11 20:32:07,732 - lava_callback - INFO - Starting async job submit for node 693b0d14fee8383e92b72156 2025-12-11 20:32:07,790 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:32:07,800 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d14fee8383e92b72156 HTTP/1.1" 502 150 2025-12-11 20:32:07,800 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0d14fee8383e92b72156'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:32:07,800 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0d14fee8383e92b72156 2025-12-11 20:32:07,801 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d14fee8383e92b72156 HTTP/1.1" 502 150 2025-12-11 20:32:07,801 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0d14fee8383e92b72156'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:32:08,606 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afabefee8383e92b6d29b 2025-12-11 20:32:08,607 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afabefee8383e92b6d29b HTTP/1.1" 502 150 2025-12-11 20:32:08,607 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afabefee8383e92b6d29b'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:32:09,801 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0d14fee8383e92b72156 2025-12-11 20:32:09,802 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d14fee8383e92b72156 HTTP/1.1" 502 150 2025-12-11 20:32:09,802 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0d14fee8383e92b72156'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:32:12,607 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afabefee8383e92b6d29b 2025-12-11 20:32:12,608 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afabefee8383e92b6d29b HTTP/1.1" 502 150 2025-12-11 20:32:12,608 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afabefee8383e92b6d29b'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:32:13,802 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0d14fee8383e92b72156 2025-12-11 20:32:13,804 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d14fee8383e92b72156 HTTP/1.1" 502 150 2025-12-11 20:32:13,804 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0d14fee8383e92b72156'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:32:14,877 - lava_callback - INFO - Received callback for node 693b1c5b15f407b7629e04de INFO: 172.21.0.1:35428 - "POST /node/693b1c5b15f407b7629e04de HTTP/1.0" 202 Accepted 2025-12-11 20:32:14,894 - lava_callback - INFO - Starting async job submit for node 693b1c5b15f407b7629e04de 2025-12-11 20:32:14,971 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:32:14,993 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c5b15f407b7629e04de HTTP/1.1" 502 150 2025-12-11 20:32:14,993 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c5b15f407b7629e04de'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:32:14,994 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c5b15f407b7629e04de 2025-12-11 20:32:14,995 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c5b15f407b7629e04de HTTP/1.1" 502 150 2025-12-11 20:32:14,995 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c5b15f407b7629e04de'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:35438 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:35440 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:32:16,996 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c5b15f407b7629e04de 2025-12-11 20:32:16,997 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c5b15f407b7629e04de HTTP/1.1" 502 150 2025-12-11 20:32:16,997 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c5b15f407b7629e04de'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:32:18,561 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9d5fee8383e92b65e8b 2025-12-11 20:32:18,563 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9d5fee8383e92b65e8b HTTP/1.1" 502 150 2025-12-11 20:32:18,563 - lava_callback - ERROR - Error processing callback for node 693ae9d5fee8383e92b65e8b: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9d5fee8383e92b65e8b (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9d5fee8383e92b65e8b (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9d5fee8383e92b65e8b (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:32:20,609 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afabefee8383e92b6d29b 2025-12-11 20:32:20,610 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afabefee8383e92b6d29b HTTP/1.1" 502 150 2025-12-11 20:32:20,610 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afabefee8383e92b6d29b'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:32:20,997 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c5b15f407b7629e04de 2025-12-11 20:32:20,998 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c5b15f407b7629e04de HTTP/1.1" 502 150 2025-12-11 20:32:20,998 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c5b15f407b7629e04de'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:32:21,805 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0d14fee8383e92b72156 2025-12-11 20:32:21,807 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d14fee8383e92b72156 HTTP/1.1" 502 150 2025-12-11 20:32:21,807 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0d14fee8383e92b72156'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:36006 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:32:27,529 - lava_callback - INFO - Received callback for node 693ae86cfee8383e92b65924 INFO: 172.21.0.1:36016 - "POST /node/693ae86cfee8383e92b65924 HTTP/1.0" 202 Accepted 2025-12-11 20:32:27,578 - lava_callback - INFO - Starting async job submit for node 693ae86cfee8383e92b65924 2025-12-11 20:32:27,603 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:32:27,636 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae86cfee8383e92b65924 HTTP/1.1" 502 150 2025-12-11 20:32:27,636 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae86cfee8383e92b65924'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:32:27,637 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae86cfee8383e92b65924 2025-12-11 20:32:27,637 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae86cfee8383e92b65924 HTTP/1.1" 502 150 2025-12-11 20:32:27,637 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae86cfee8383e92b65924'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:36030 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:32:28,999 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c5b15f407b7629e04de 2025-12-11 20:32:28,999 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c5b15f407b7629e04de HTTP/1.1" 502 150 2025-12-11 20:32:29,000 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1c5b15f407b7629e04de'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:32:29,638 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae86cfee8383e92b65924 2025-12-11 20:32:29,639 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae86cfee8383e92b65924 HTTP/1.1" 502 150 2025-12-11 20:32:29,639 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae86cfee8383e92b65924'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:32:33,639 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae86cfee8383e92b65924 2025-12-11 20:32:33,640 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae86cfee8383e92b65924 HTTP/1.1" 502 150 2025-12-11 20:32:33,640 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae86cfee8383e92b65924'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:34154 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:32:36,610 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afabefee8383e92b6d29b 2025-12-11 20:32:36,612 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afabefee8383e92b6d29b HTTP/1.1" 502 150 2025-12-11 20:32:36,612 - lava_callback - ERROR - Error processing callback for node 693afabefee8383e92b6d29b: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afabefee8383e92b6d29b (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afabefee8383e92b6d29b (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afabefee8383e92b6d29b (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:32:37,807 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0d14fee8383e92b72156 2025-12-11 20:32:37,808 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d14fee8383e92b72156 HTTP/1.1" 502 150 2025-12-11 20:32:37,808 - lava_callback - ERROR - Error processing callback for node 693b0d14fee8383e92b72156: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0d14fee8383e92b72156 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0d14fee8383e92b72156 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0d14fee8383e92b72156 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:32:41,641 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae86cfee8383e92b65924 2025-12-11 20:32:41,642 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae86cfee8383e92b65924 HTTP/1.1" 502 150 2025-12-11 20:32:41,642 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae86cfee8383e92b65924'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:32:45,000 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1c5b15f407b7629e04de 2025-12-11 20:32:45,001 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1c5b15f407b7629e04de HTTP/1.1" 502 150 2025-12-11 20:32:45,001 - lava_callback - ERROR - Error processing callback for node 693b1c5b15f407b7629e04de: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1c5b15f407b7629e04de (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1c5b15f407b7629e04de (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1c5b15f407b7629e04de (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:41684 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:32:52,612 - lava_callback - INFO - Received callback for node 693b181c15f407b7629df3fc INFO: 172.21.0.1:56322 - "POST /node/693b181c15f407b7629df3fc HTTP/1.0" 202 Accepted 2025-12-11 20:32:52,626 - lava_callback - INFO - Starting async job submit for node 693b181c15f407b7629df3fc 2025-12-11 20:32:52,646 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:32:52,675 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b181c15f407b7629df3fc HTTP/1.1" 502 150 2025-12-11 20:32:52,675 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b181c15f407b7629df3fc'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:32:52,675 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b181c15f407b7629df3fc 2025-12-11 20:32:52,675 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b181c15f407b7629df3fc HTTP/1.1" 502 150 2025-12-11 20:32:52,675 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b181c15f407b7629df3fc'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:32:54,676 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b181c15f407b7629df3fc 2025-12-11 20:32:54,677 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b181c15f407b7629df3fc HTTP/1.1" 502 150 2025-12-11 20:32:54,677 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b181c15f407b7629df3fc'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:56338 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:32:57,642 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae86cfee8383e92b65924 2025-12-11 20:32:57,643 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae86cfee8383e92b65924 HTTP/1.1" 502 150 2025-12-11 20:32:57,644 - lava_callback - ERROR - Error processing callback for node 693ae86cfee8383e92b65924: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae86cfee8383e92b65924 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae86cfee8383e92b65924 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae86cfee8383e92b65924 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:32:58,677 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b181c15f407b7629df3fc 2025-12-11 20:32:58,678 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b181c15f407b7629df3fc HTTP/1.1" 502 150 2025-12-11 20:32:58,679 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b181c15f407b7629df3fc'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:46828 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:33:06,679 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b181c15f407b7629df3fc 2025-12-11 20:33:06,680 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b181c15f407b7629df3fc HTTP/1.1" 502 150 2025-12-11 20:33:06,680 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b181c15f407b7629df3fc'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:33:09,428 - lava_callback - INFO - Received callback for node 693b0985fee8383e92b717a6 INFO: 172.21.0.1:46834 - "POST /node/693b0985fee8383e92b717a6 HTTP/1.0" 202 Accepted 2025-12-11 20:33:09,436 - lava_callback - INFO - Starting async job submit for node 693b0985fee8383e92b717a6 2025-12-11 20:33:09,498 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:33:09,512 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0985fee8383e92b717a6 HTTP/1.1" 502 150 2025-12-11 20:33:09,513 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0985fee8383e92b717a6'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:33:09,513 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0985fee8383e92b717a6 2025-12-11 20:33:09,513 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0985fee8383e92b717a6 HTTP/1.1" 502 150 2025-12-11 20:33:09,513 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0985fee8383e92b717a6'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:33:11,514 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0985fee8383e92b717a6 2025-12-11 20:33:11,515 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0985fee8383e92b717a6 HTTP/1.1" 502 150 2025-12-11 20:33:11,515 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0985fee8383e92b717a6'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:44854 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 20:33:15,516 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0985fee8383e92b717a6 2025-12-11 20:33:15,517 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0985fee8383e92b717a6 HTTP/1.1" 502 150 2025-12-11 20:33:15,517 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0985fee8383e92b717a6'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:44856 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:33:21,517 - lava_callback - INFO - Received callback for node 693b02cffee8383e92b6f1d3 INFO: 172.21.0.1:45632 - "POST /node/693b02cffee8383e92b6f1d3 HTTP/1.0" 202 Accepted 2025-12-11 20:33:21,525 - lava_callback - INFO - Starting async job submit for node 693b02cffee8383e92b6f1d3 2025-12-11 20:33:21,588 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:33:21,620 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b02cffee8383e92b6f1d3 HTTP/1.1" 502 150 2025-12-11 20:33:21,621 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b02cffee8383e92b6f1d3'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:33:21,621 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b02cffee8383e92b6f1d3 2025-12-11 20:33:21,626 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b02cffee8383e92b6f1d3 HTTP/1.1" 502 150 2025-12-11 20:33:21,626 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b02cffee8383e92b6f1d3'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:33:22,680 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b181c15f407b7629df3fc 2025-12-11 20:33:22,682 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b181c15f407b7629df3fc HTTP/1.1" 502 150 2025-12-11 20:33:22,682 - lava_callback - ERROR - Error processing callback for node 693b181c15f407b7629df3fc: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b181c15f407b7629df3fc (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b181c15f407b7629df3fc (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b181c15f407b7629df3fc (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:33:23,517 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0985fee8383e92b717a6 2025-12-11 20:33:23,518 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0985fee8383e92b717a6 HTTP/1.1" 502 150 2025-12-11 20:33:23,518 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0985fee8383e92b717a6'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:33:23,628 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b02cffee8383e92b6f1d3 2025-12-11 20:33:23,629 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b02cffee8383e92b6f1d3 HTTP/1.1" 502 150 2025-12-11 20:33:23,630 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b02cffee8383e92b6f1d3'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:45648 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:33:26,620 - lava_callback - INFO - Received callback for node 693b1add15f407b7629dff75 INFO: 172.21.0.1:45656 - "POST /node/693b1add15f407b7629dff75 HTTP/1.0" 202 Accepted 2025-12-11 20:33:26,626 - lava_callback - INFO - Starting async job submit for node 693b1add15f407b7629dff75 2025-12-11 20:33:26,675 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:33:26,689 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1add15f407b7629dff75 HTTP/1.1" 502 150 2025-12-11 20:33:26,689 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1add15f407b7629dff75'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:33:26,689 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1add15f407b7629dff75 2025-12-11 20:33:26,690 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1add15f407b7629dff75 HTTP/1.1" 502 150 2025-12-11 20:33:26,690 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1add15f407b7629dff75'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:33:27,630 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b02cffee8383e92b6f1d3 2025-12-11 20:33:27,631 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b02cffee8383e92b6f1d3 HTTP/1.1" 502 150 2025-12-11 20:33:27,631 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b02cffee8383e92b6f1d3'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:45664 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:33:28,691 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1add15f407b7629dff75 2025-12-11 20:33:28,697 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1add15f407b7629dff75 HTTP/1.1" 502 150 2025-12-11 20:33:28,697 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1add15f407b7629dff75'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:33:29,872 - lava_callback - INFO - Received callback for node 693b0271fee8383e92b6f042 INFO: 172.21.0.1:44568 - "POST /node/693b0271fee8383e92b6f042 HTTP/1.0" 202 Accepted 2025-12-11 20:33:29,884 - lava_callback - INFO - Starting async job submit for node 693b0271fee8383e92b6f042 2025-12-11 20:33:29,983 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:33:29,994 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0271fee8383e92b6f042 HTTP/1.1" 502 150 2025-12-11 20:33:29,994 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0271fee8383e92b6f042'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:33:29,994 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0271fee8383e92b6f042 2025-12-11 20:33:29,994 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0271fee8383e92b6f042 HTTP/1.1" 502 150 2025-12-11 20:33:29,994 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0271fee8383e92b6f042'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:33:31,995 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0271fee8383e92b6f042 2025-12-11 20:33:32,000 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0271fee8383e92b6f042 HTTP/1.1" 502 150 2025-12-11 20:33:32,000 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0271fee8383e92b6f042'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:33:32,698 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1add15f407b7629dff75 2025-12-11 20:33:32,699 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1add15f407b7629dff75 HTTP/1.1" 502 150 2025-12-11 20:33:32,699 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1add15f407b7629dff75'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:33:35,631 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b02cffee8383e92b6f1d3 2025-12-11 20:33:35,632 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b02cffee8383e92b6f1d3 HTTP/1.1" 502 150 2025-12-11 20:33:35,632 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b02cffee8383e92b6f1d3'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:33:36,000 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0271fee8383e92b6f042 2025-12-11 20:33:36,006 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0271fee8383e92b6f042 HTTP/1.1" 502 150 2025-12-11 20:33:36,007 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0271fee8383e92b6f042'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:44572 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:33:39,519 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0985fee8383e92b717a6 2025-12-11 20:33:39,520 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0985fee8383e92b717a6 HTTP/1.1" 502 150 2025-12-11 20:33:39,520 - lava_callback - ERROR - Error processing callback for node 693b0985fee8383e92b717a6: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0985fee8383e92b717a6 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0985fee8383e92b717a6 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0985fee8383e92b717a6 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:33:40,699 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1add15f407b7629dff75 2025-12-11 20:33:40,700 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1add15f407b7629dff75 HTTP/1.1" 502 150 2025-12-11 20:33:40,700 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1add15f407b7629dff75'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:33:44,007 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0271fee8383e92b6f042 2025-12-11 20:33:44,008 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0271fee8383e92b6f042 HTTP/1.1" 502 150 2025-12-11 20:33:44,009 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0271fee8383e92b6f042'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:37326 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:33:51,633 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b02cffee8383e92b6f1d3 2025-12-11 20:33:51,634 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b02cffee8383e92b6f1d3 HTTP/1.1" 502 150 2025-12-11 20:33:51,634 - lava_callback - ERROR - Error processing callback for node 693b02cffee8383e92b6f1d3: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b02cffee8383e92b6f1d3 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b02cffee8383e92b6f1d3 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b02cffee8383e92b6f1d3 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:32888 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:33:56,701 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1add15f407b7629dff75 2025-12-11 20:33:56,702 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1add15f407b7629dff75 HTTP/1.1" 502 150 2025-12-11 20:33:56,702 - lava_callback - ERROR - Error processing callback for node 693b1add15f407b7629dff75: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1add15f407b7629dff75 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1add15f407b7629dff75 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1add15f407b7629dff75 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:34:00,009 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0271fee8383e92b6f042 2025-12-11 20:34:00,010 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0271fee8383e92b6f042 HTTP/1.1" 502 150 2025-12-11 20:34:00,010 - lava_callback - ERROR - Error processing callback for node 693b0271fee8383e92b6f042: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0271fee8383e92b6f042 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0271fee8383e92b6f042 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0271fee8383e92b6f042 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:54528 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:34:14,777 - lava_callback - INFO - Received callback for node 693b0d68fee8383e92b721f8 INFO: 172.21.0.1:49408 - "POST /node/693b0d68fee8383e92b721f8 HTTP/1.0" 202 Accepted 2025-12-11 20:34:14,783 - lava_callback - INFO - Starting async job submit for node 693b0d68fee8383e92b721f8 2025-12-11 20:34:14,822 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:34:14,856 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d68fee8383e92b721f8 HTTP/1.1" 502 150 2025-12-11 20:34:14,856 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0d68fee8383e92b721f8'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:34:14,856 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0d68fee8383e92b721f8 2025-12-11 20:34:14,857 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d68fee8383e92b721f8 HTTP/1.1" 502 150 2025-12-11 20:34:14,857 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0d68fee8383e92b721f8'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:49416 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:49418 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:34:16,857 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0d68fee8383e92b721f8 2025-12-11 20:34:16,858 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d68fee8383e92b721f8 HTTP/1.1" 502 150 2025-12-11 20:34:16,858 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0d68fee8383e92b721f8'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:34:20,858 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0d68fee8383e92b721f8 2025-12-11 20:34:20,860 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d68fee8383e92b721f8 HTTP/1.1" 502 150 2025-12-11 20:34:20,860 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0d68fee8383e92b721f8'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:52218 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52224 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:34:28,860 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0d68fee8383e92b721f8 2025-12-11 20:34:28,861 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d68fee8383e92b721f8 HTTP/1.1" 502 150 2025-12-11 20:34:28,862 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0d68fee8383e92b721f8'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:57222 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:34:44,862 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0d68fee8383e92b721f8 2025-12-11 20:34:44,863 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d68fee8383e92b721f8 HTTP/1.1" 502 150 2025-12-11 20:34:44,863 - lava_callback - ERROR - Error processing callback for node 693b0d68fee8383e92b721f8: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0d68fee8383e92b721f8 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0d68fee8383e92b721f8 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0d68fee8383e92b721f8 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:42490 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:34:46,806 - lava_callback - INFO - Received callback for node 693b0eedfee8383e92b72943 INFO: 172.21.0.1:42494 - "POST /node/693b0eedfee8383e92b72943 HTTP/1.0" 202 Accepted 2025-12-11 20:34:46,812 - lava_callback - INFO - Starting async job submit for node 693b0eedfee8383e92b72943 2025-12-11 20:34:46,849 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:34:46,859 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0eedfee8383e92b72943 HTTP/1.1" 502 150 2025-12-11 20:34:46,859 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0eedfee8383e92b72943'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:34:46,859 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0eedfee8383e92b72943 2025-12-11 20:34:46,860 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0eedfee8383e92b72943 HTTP/1.1" 502 150 2025-12-11 20:34:46,860 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0eedfee8383e92b72943'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:34:48,860 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0eedfee8383e92b72943 2025-12-11 20:34:48,861 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0eedfee8383e92b72943 HTTP/1.1" 502 150 2025-12-11 20:34:48,861 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0eedfee8383e92b72943'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:34:52,862 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0eedfee8383e92b72943 2025-12-11 20:34:52,863 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0eedfee8383e92b72943 HTTP/1.1" 502 150 2025-12-11 20:34:52,863 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0eedfee8383e92b72943'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:48504 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:35:00,863 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0eedfee8383e92b72943 2025-12-11 20:35:00,864 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0eedfee8383e92b72943 HTTP/1.1" 502 150 2025-12-11 20:35:00,864 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0eedfee8383e92b72943'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:57824 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:35:06,885 - lava_callback - INFO - Received callback for node 693afaddfee8383e92b6d2e7 INFO: 172.21.0.1:57836 - "POST /node/693afaddfee8383e92b6d2e7 HTTP/1.0" 202 Accepted 2025-12-11 20:35:06,891 - lava_callback - INFO - Starting async job submit for node 693afaddfee8383e92b6d2e7 2025-12-11 20:35:06,909 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:35:06,922 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afaddfee8383e92b6d2e7 HTTP/1.1" 502 150 2025-12-11 20:35:06,922 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afaddfee8383e92b6d2e7'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:35:06,923 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afaddfee8383e92b6d2e7 2025-12-11 20:35:06,923 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afaddfee8383e92b6d2e7 HTTP/1.1" 502 150 2025-12-11 20:35:06,923 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afaddfee8383e92b6d2e7'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:35:08,924 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afaddfee8383e92b6d2e7 2025-12-11 20:35:08,925 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afaddfee8383e92b6d2e7 HTTP/1.1" 502 150 2025-12-11 20:35:08,925 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afaddfee8383e92b6d2e7'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:35:12,925 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afaddfee8383e92b6d2e7 2025-12-11 20:35:12,927 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afaddfee8383e92b6d2e7 HTTP/1.1" 502 150 2025-12-11 20:35:12,927 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afaddfee8383e92b6d2e7'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:38714 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:38728 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:35:16,864 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0eedfee8383e92b72943 2025-12-11 20:35:16,870 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0eedfee8383e92b72943 HTTP/1.1" 502 150 2025-12-11 20:35:16,870 - lava_callback - ERROR - Error processing callback for node 693b0eedfee8383e92b72943: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0eedfee8383e92b72943 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0eedfee8383e92b72943 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0eedfee8383e92b72943 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:35:20,927 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afaddfee8383e92b6d2e7 2025-12-11 20:35:20,928 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afaddfee8383e92b6d2e7 HTTP/1.1" 502 150 2025-12-11 20:35:20,928 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afaddfee8383e92b6d2e7'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:35354 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35364 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33074 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:35:36,928 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afaddfee8383e92b6d2e7 2025-12-11 20:35:36,929 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afaddfee8383e92b6d2e7 HTTP/1.1" 502 150 2025-12-11 20:35:36,930 - lava_callback - ERROR - Error processing callback for node 693afaddfee8383e92b6d2e7: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afaddfee8383e92b6d2e7 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afaddfee8383e92b6d2e7 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afaddfee8383e92b6d2e7 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:35:40,839 - lava_callback - INFO - Received callback for node 693b0e3efee8383e92b72572 INFO: 172.21.0.1:47994 - "POST /node/693b0e3efee8383e92b72572 HTTP/1.0" 202 Accepted 2025-12-11 20:35:40,850 - lava_callback - INFO - Starting async job submit for node 693b0e3efee8383e92b72572 2025-12-11 20:35:40,892 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:35:40,904 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e3efee8383e92b72572 HTTP/1.1" 502 150 2025-12-11 20:35:40,904 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e3efee8383e92b72572'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:35:40,904 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e3efee8383e92b72572 2025-12-11 20:35:40,906 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e3efee8383e92b72572 HTTP/1.1" 502 150 2025-12-11 20:35:40,906 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e3efee8383e92b72572'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:35:42,906 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e3efee8383e92b72572 2025-12-11 20:35:42,908 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e3efee8383e92b72572 HTTP/1.1" 502 150 2025-12-11 20:35:42,908 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e3efee8383e92b72572'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:35:43,790 - lava_callback - INFO - Received callback for node 693b0d68fee8383e92b721fc INFO: 172.21.0.1:48004 - "POST /node/693b0d68fee8383e92b721fc HTTP/1.0" 202 Accepted 2025-12-11 20:35:43,797 - lava_callback - INFO - Starting async job submit for node 693b0d68fee8383e92b721fc 2025-12-11 20:35:43,818 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:35:43,828 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d68fee8383e92b721fc HTTP/1.1" 502 150 2025-12-11 20:35:43,828 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0d68fee8383e92b721fc'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:35:43,828 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0d68fee8383e92b721fc 2025-12-11 20:35:43,829 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d68fee8383e92b721fc HTTP/1.1" 502 150 2025-12-11 20:35:43,829 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0d68fee8383e92b721fc'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:35:45,829 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0d68fee8383e92b721fc 2025-12-11 20:35:45,830 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d68fee8383e92b721fc HTTP/1.1" 502 150 2025-12-11 20:35:45,830 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0d68fee8383e92b721fc'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:48016 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:35:46,911 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e3efee8383e92b72572 2025-12-11 20:35:46,914 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e3efee8383e92b72572 HTTP/1.1" 502 150 2025-12-11 20:35:46,914 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e3efee8383e92b72572'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:35:49,830 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0d68fee8383e92b721fc 2025-12-11 20:35:49,832 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d68fee8383e92b721fc HTTP/1.1" 502 150 2025-12-11 20:35:49,832 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0d68fee8383e92b721fc'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:35:52,936 - lava_callback - INFO - Received callback for node 693b181c15f407b7629df3ff INFO: 172.21.0.1:36900 - "POST /node/693b181c15f407b7629df3ff HTTP/1.0" 202 Accepted 2025-12-11 20:35:52,944 - lava_callback - INFO - Starting async job submit for node 693b181c15f407b7629df3ff 2025-12-11 20:35:52,963 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:35:52,975 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b181c15f407b7629df3ff HTTP/1.1" 502 150 2025-12-11 20:35:52,976 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b181c15f407b7629df3ff'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:35:52,976 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b181c15f407b7629df3ff 2025-12-11 20:35:52,976 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b181c15f407b7629df3ff HTTP/1.1" 502 150 2025-12-11 20:35:52,976 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b181c15f407b7629df3ff'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:35:54,914 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e3efee8383e92b72572 2025-12-11 20:35:54,916 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e3efee8383e92b72572 HTTP/1.1" 502 150 2025-12-11 20:35:54,916 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e3efee8383e92b72572'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:35:54,977 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b181c15f407b7629df3ff 2025-12-11 20:35:54,978 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b181c15f407b7629df3ff HTTP/1.1" 502 150 2025-12-11 20:35:54,978 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b181c15f407b7629df3ff'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:36908 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:35:57,832 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0d68fee8383e92b721fc 2025-12-11 20:35:57,833 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d68fee8383e92b721fc HTTP/1.1" 502 150 2025-12-11 20:35:57,833 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0d68fee8383e92b721fc'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:35:58,978 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b181c15f407b7629df3ff 2025-12-11 20:35:58,979 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b181c15f407b7629df3ff HTTP/1.1" 502 150 2025-12-11 20:35:58,979 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b181c15f407b7629df3ff'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:41316 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:36:06,715 - lava_callback - INFO - Received callback for node 693b0ef4fee8383e92b7296e INFO: 172.21.0.1:41324 - "POST /node/693b0ef4fee8383e92b7296e HTTP/1.0" 202 Accepted 2025-12-11 20:36:06,722 - lava_callback - INFO - Starting async job submit for node 693b0ef4fee8383e92b7296e 2025-12-11 20:36:06,746 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:36:06,765 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0ef4fee8383e92b7296e HTTP/1.1" 502 150 2025-12-11 20:36:06,766 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0ef4fee8383e92b7296e'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:06,766 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0ef4fee8383e92b7296e 2025-12-11 20:36:06,766 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0ef4fee8383e92b7296e HTTP/1.1" 502 150 2025-12-11 20:36:06,766 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0ef4fee8383e92b7296e'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:06,980 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b181c15f407b7629df3ff 2025-12-11 20:36:06,981 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b181c15f407b7629df3ff HTTP/1.1" 502 150 2025-12-11 20:36:06,981 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b181c15f407b7629df3ff'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:08,767 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0ef4fee8383e92b7296e 2025-12-11 20:36:08,768 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0ef4fee8383e92b7296e HTTP/1.1" 502 150 2025-12-11 20:36:08,768 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0ef4fee8383e92b7296e'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:10,916 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e3efee8383e92b72572 2025-12-11 20:36:10,917 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e3efee8383e92b72572 HTTP/1.1" 502 150 2025-12-11 20:36:10,918 - lava_callback - ERROR - Error processing callback for node 693b0e3efee8383e92b72572: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e3efee8383e92b72572 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e3efee8383e92b72572 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e3efee8383e92b72572 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:36:12,768 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0ef4fee8383e92b7296e 2025-12-11 20:36:12,770 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0ef4fee8383e92b7296e HTTP/1.1" 502 150 2025-12-11 20:36:12,770 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0ef4fee8383e92b7296e'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:13,833 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0d68fee8383e92b721fc 2025-12-11 20:36:13,835 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d68fee8383e92b721fc HTTP/1.1" 502 150 2025-12-11 20:36:13,835 - lava_callback - ERROR - Error processing callback for node 693b0d68fee8383e92b721fc: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0d68fee8383e92b721fc (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0d68fee8383e92b721fc (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0d68fee8383e92b721fc (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:52086 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:52096 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:36:20,770 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0ef4fee8383e92b7296e 2025-12-11 20:36:20,771 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0ef4fee8383e92b7296e HTTP/1.1" 502 150 2025-12-11 20:36:20,771 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0ef4fee8383e92b7296e'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:22,981 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b181c15f407b7629df3ff 2025-12-11 20:36:22,990 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b181c15f407b7629df3ff HTTP/1.1" 502 150 2025-12-11 20:36:22,990 - lava_callback - ERROR - Error processing callback for node 693b181c15f407b7629df3ff: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b181c15f407b7629df3ff (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b181c15f407b7629df3ff (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b181c15f407b7629df3ff (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:36:25,992 - lava_callback - INFO - Received callback for node 6938883e63d980a9748048b2 INFO: 172.21.0.1:47278 - "POST /node/6938883e63d980a9748048b2 HTTP/1.0" 202 Accepted 2025-12-11 20:36:26,001 - lava_callback - INFO - Starting async job submit for node 6938883e63d980a9748048b2 2025-12-11 20:36:26,003 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:36:26,037 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938883e63d980a9748048b2 HTTP/1.1" 502 150 2025-12-11 20:36:26,037 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938883e63d980a9748048b2'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:26,037 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938883e63d980a9748048b2 2025-12-11 20:36:26,038 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938883e63d980a9748048b2 HTTP/1.1" 502 150 2025-12-11 20:36:26,038 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938883e63d980a9748048b2'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:26,141 - lava_callback - INFO - Received callback for node 693b02dafee8383e92b6f1ff INFO: 172.21.0.1:47292 - "POST /node/693b02dafee8383e92b6f1ff HTTP/1.0" 202 Accepted 2025-12-11 20:36:26,153 - lava_callback - INFO - Starting async job submit for node 693b02dafee8383e92b6f1ff 2025-12-11 20:36:26,248 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:36:26,264 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b02dafee8383e92b6f1ff HTTP/1.1" 502 150 2025-12-11 20:36:26,265 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b02dafee8383e92b6f1ff'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:26,265 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b02dafee8383e92b6f1ff 2025-12-11 20:36:26,265 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b02dafee8383e92b6f1ff HTTP/1.1" 502 150 2025-12-11 20:36:26,266 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b02dafee8383e92b6f1ff'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:47306 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:36:27,692 - lava_callback - INFO - Received callback for node 693ae9ecfee8383e92b65edc INFO: 172.21.0.1:47310 - "POST /node/693ae9ecfee8383e92b65edc HTTP/1.0" 202 Accepted 2025-12-11 20:36:27,699 - lava_callback - INFO - Starting async job submit for node 693ae9ecfee8383e92b65edc 2025-12-11 20:36:27,725 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:36:27,768 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9ecfee8383e92b65edc HTTP/1.1" 502 150 2025-12-11 20:36:27,768 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9ecfee8383e92b65edc'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:27,768 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9ecfee8383e92b65edc 2025-12-11 20:36:27,769 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9ecfee8383e92b65edc HTTP/1.1" 502 150 2025-12-11 20:36:27,769 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9ecfee8383e92b65edc'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:47316 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:36:28,038 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938883e63d980a9748048b2 2025-12-11 20:36:28,040 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938883e63d980a9748048b2 HTTP/1.1" 502 150 2025-12-11 20:36:28,040 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938883e63d980a9748048b2'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:28,266 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b02dafee8383e92b6f1ff 2025-12-11 20:36:28,267 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b02dafee8383e92b6f1ff HTTP/1.1" 502 150 2025-12-11 20:36:28,267 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b02dafee8383e92b6f1ff'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:29,770 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9ecfee8383e92b65edc 2025-12-11 20:36:29,771 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9ecfee8383e92b65edc HTTP/1.1" 502 150 2025-12-11 20:36:29,771 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9ecfee8383e92b65edc'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:30,400 - lava_callback - INFO - Received callback for node 693b0e41fee8383e92b72581 INFO: 172.21.0.1:33176 - "POST /node/693b0e41fee8383e92b72581 HTTP/1.0" 202 Accepted 2025-12-11 20:36:30,407 - lava_callback - INFO - Starting async job submit for node 693b0e41fee8383e92b72581 2025-12-11 20:36:30,435 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:36:30,445 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e41fee8383e92b72581 HTTP/1.1" 502 150 2025-12-11 20:36:30,445 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e41fee8383e92b72581'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:30,445 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e41fee8383e92b72581 2025-12-11 20:36:30,446 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e41fee8383e92b72581 HTTP/1.1" 502 150 2025-12-11 20:36:30,446 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e41fee8383e92b72581'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:32,040 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938883e63d980a9748048b2 2025-12-11 20:36:32,041 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938883e63d980a9748048b2 HTTP/1.1" 502 150 2025-12-11 20:36:32,041 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938883e63d980a9748048b2'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:32,267 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b02dafee8383e92b6f1ff 2025-12-11 20:36:32,269 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b02dafee8383e92b6f1ff HTTP/1.1" 502 150 2025-12-11 20:36:32,269 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b02dafee8383e92b6f1ff'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:32,446 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e41fee8383e92b72581 2025-12-11 20:36:32,448 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e41fee8383e92b72581 HTTP/1.1" 502 150 2025-12-11 20:36:32,448 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e41fee8383e92b72581'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:33,772 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9ecfee8383e92b65edc 2025-12-11 20:36:33,773 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9ecfee8383e92b65edc HTTP/1.1" 502 150 2025-12-11 20:36:33,773 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9ecfee8383e92b65edc'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:36,448 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e41fee8383e92b72581 2025-12-11 20:36:36,449 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e41fee8383e92b72581 HTTP/1.1" 502 150 2025-12-11 20:36:36,449 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e41fee8383e92b72581'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:33192 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:36:36,772 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0ef4fee8383e92b7296e 2025-12-11 20:36:36,773 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0ef4fee8383e92b7296e HTTP/1.1" 502 150 2025-12-11 20:36:36,773 - lava_callback - ERROR - Error processing callback for node 693b0ef4fee8383e92b7296e: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0ef4fee8383e92b7296e (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0ef4fee8383e92b7296e (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0ef4fee8383e92b7296e (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:36:40,041 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938883e63d980a9748048b2 2025-12-11 20:36:40,043 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938883e63d980a9748048b2 HTTP/1.1" 502 150 2025-12-11 20:36:40,043 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938883e63d980a9748048b2'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:40,269 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b02dafee8383e92b6f1ff 2025-12-11 20:36:40,271 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b02dafee8383e92b6f1ff HTTP/1.1" 502 150 2025-12-11 20:36:40,271 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b02dafee8383e92b6f1ff'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:41,773 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9ecfee8383e92b65edc 2025-12-11 20:36:41,774 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9ecfee8383e92b65edc HTTP/1.1" 502 150 2025-12-11 20:36:41,774 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9ecfee8383e92b65edc'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:44,449 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e41fee8383e92b72581 2025-12-11 20:36:44,451 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e41fee8383e92b72581 HTTP/1.1" 502 150 2025-12-11 20:36:44,451 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e41fee8383e92b72581'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:36360 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:36:48,222 - lava_callback - INFO - Received callback for node 693ae9dafee8383e92b65e9e INFO: 172.21.0.1:36368 - "POST /node/693ae9dafee8383e92b65e9e HTTP/1.0" 202 Accepted 2025-12-11 20:36:48,236 - lava_callback - INFO - Starting async job submit for node 693ae9dafee8383e92b65e9e 2025-12-11 20:36:48,594 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:36:48,604 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9dafee8383e92b65e9e HTTP/1.1" 502 150 2025-12-11 20:36:48,604 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9dafee8383e92b65e9e'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:48,604 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9dafee8383e92b65e9e 2025-12-11 20:36:48,605 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9dafee8383e92b65e9e HTTP/1.1" 502 150 2025-12-11 20:36:48,605 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9dafee8383e92b65e9e'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:50,605 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9dafee8383e92b65e9e 2025-12-11 20:36:50,606 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9dafee8383e92b65e9e HTTP/1.1" 502 150 2025-12-11 20:36:50,606 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9dafee8383e92b65e9e'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:54,607 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9dafee8383e92b65e9e 2025-12-11 20:36:54,610 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9dafee8383e92b65e9e HTTP/1.1" 502 150 2025-12-11 20:36:54,610 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9dafee8383e92b65e9e'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:54,734 - lava_callback - INFO - Received callback for node 693b0d68fee8383e92b721f9 INFO: 172.21.0.1:35556 - "POST /node/693b0d68fee8383e92b721f9 HTTP/1.0" 202 Accepted 2025-12-11 20:36:54,750 - lava_callback - INFO - Starting async job submit for node 693b0d68fee8383e92b721f9 2025-12-11 20:36:54,856 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:36:54,883 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d68fee8383e92b721f9 HTTP/1.1" 502 150 2025-12-11 20:36:54,883 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0d68fee8383e92b721f9'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:54,883 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0d68fee8383e92b721f9 2025-12-11 20:36:54,885 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d68fee8383e92b721f9 HTTP/1.1" 502 150 2025-12-11 20:36:54,886 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0d68fee8383e92b721f9'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:56,043 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938883e63d980a9748048b2 2025-12-11 20:36:56,044 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938883e63d980a9748048b2 HTTP/1.1" 502 150 2025-12-11 20:36:56,045 - lava_callback - ERROR - Error processing callback for node 6938883e63d980a9748048b2: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938883e63d980a9748048b2 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938883e63d980a9748048b2 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938883e63d980a9748048b2 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:36:56,271 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b02dafee8383e92b6f1ff 2025-12-11 20:36:56,272 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b02dafee8383e92b6f1ff HTTP/1.1" 502 150 2025-12-11 20:36:56,272 - lava_callback - ERROR - Error processing callback for node 693b02dafee8383e92b6f1ff: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b02dafee8383e92b6f1ff (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b02dafee8383e92b6f1ff (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b02dafee8383e92b6f1ff (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:35572 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:36:56,886 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0d68fee8383e92b721f9 2025-12-11 20:36:56,887 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d68fee8383e92b721f9 HTTP/1.1" 502 150 2025-12-11 20:36:56,887 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0d68fee8383e92b721f9'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:36:57,775 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9ecfee8383e92b65edc 2025-12-11 20:36:57,776 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9ecfee8383e92b65edc HTTP/1.1" 502 150 2025-12-11 20:36:57,776 - lava_callback - ERROR - Error processing callback for node 693ae9ecfee8383e92b65edc: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9ecfee8383e92b65edc (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9ecfee8383e92b65edc (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9ecfee8383e92b65edc (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:37:00,451 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e41fee8383e92b72581 2025-12-11 20:37:00,452 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e41fee8383e92b72581 HTTP/1.1" 502 150 2025-12-11 20:37:00,453 - lava_callback - ERROR - Error processing callback for node 693b0e41fee8383e92b72581: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e41fee8383e92b72581 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e41fee8383e92b72581 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e41fee8383e92b72581 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:37:00,887 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0d68fee8383e92b721f9 2025-12-11 20:37:00,889 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d68fee8383e92b721f9 HTTP/1.1" 502 150 2025-12-11 20:37:00,889 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0d68fee8383e92b721f9'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:37:02,611 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9dafee8383e92b65e9e 2025-12-11 20:37:02,612 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9dafee8383e92b65e9e HTTP/1.1" 502 150 2025-12-11 20:37:02,612 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9dafee8383e92b65e9e'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:37:06,454 - lava_callback - INFO - Received callback for node 6938887163d980a9748048be INFO: 172.21.0.1:37510 - "POST /node/6938887163d980a9748048be HTTP/1.0" 202 Accepted 2025-12-11 20:37:06,460 - lava_callback - INFO - Starting async job submit for node 6938887163d980a9748048be 2025-12-11 20:37:06,460 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:37:06,493 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938887163d980a9748048be HTTP/1.1" 502 150 2025-12-11 20:37:06,493 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938887163d980a9748048be'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:37:06,493 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938887163d980a9748048be 2025-12-11 20:37:06,494 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938887163d980a9748048be HTTP/1.1" 502 150 2025-12-11 20:37:06,494 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938887163d980a9748048be'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:37518 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:37:06,977 - lava_callback - INFO - Received callback for node 693b197d15f407b7629df654 INFO: 172.21.0.1:37532 - "POST /node/693b197d15f407b7629df654 HTTP/1.0" 202 Accepted 2025-12-11 20:37:06,984 - lava_callback - INFO - Starting async job submit for node 693b197d15f407b7629df654 2025-12-11 20:37:07,019 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:37:07,029 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b197d15f407b7629df654 HTTP/1.1" 502 150 2025-12-11 20:37:07,029 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b197d15f407b7629df654'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:37:07,029 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b197d15f407b7629df654 2025-12-11 20:37:07,029 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b197d15f407b7629df654 HTTP/1.1" 502 150 2025-12-11 20:37:07,029 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b197d15f407b7629df654'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:37:08,494 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938887163d980a9748048be 2025-12-11 20:37:08,495 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938887163d980a9748048be HTTP/1.1" 502 150 2025-12-11 20:37:08,495 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938887163d980a9748048be'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:37:08,889 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0d68fee8383e92b721f9 2025-12-11 20:37:08,890 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d68fee8383e92b721f9 HTTP/1.1" 502 150 2025-12-11 20:37:08,890 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0d68fee8383e92b721f9'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:37:09,030 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b197d15f407b7629df654 2025-12-11 20:37:09,031 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b197d15f407b7629df654 HTTP/1.1" 502 150 2025-12-11 20:37:09,031 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b197d15f407b7629df654'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:37:12,495 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938887163d980a9748048be 2025-12-11 20:37:12,496 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938887163d980a9748048be HTTP/1.1" 502 150 2025-12-11 20:37:12,497 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938887163d980a9748048be'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:37:13,031 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b197d15f407b7629df654 2025-12-11 20:37:13,033 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b197d15f407b7629df654 HTTP/1.1" 502 150 2025-12-11 20:37:13,033 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b197d15f407b7629df654'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:50690 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:50702 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:37:17,594 - lava_callback - INFO - Received callback for node 693b0e42fee8383e92b72591 INFO: 172.21.0.1:50712 - "POST /node/693b0e42fee8383e92b72591 HTTP/1.0" 202 Accepted 2025-12-11 20:37:17,601 - lava_callback - INFO - Starting async job submit for node 693b0e42fee8383e92b72591 2025-12-11 20:37:17,630 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:37:17,639 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e42fee8383e92b72591 HTTP/1.1" 502 150 2025-12-11 20:37:17,640 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e42fee8383e92b72591'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:37:17,640 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e42fee8383e92b72591 2025-12-11 20:37:17,640 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e42fee8383e92b72591 HTTP/1.1" 502 150 2025-12-11 20:37:17,640 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e42fee8383e92b72591'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:37:18,612 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9dafee8383e92b65e9e 2025-12-11 20:37:18,613 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9dafee8383e92b65e9e HTTP/1.1" 502 150 2025-12-11 20:37:18,614 - lava_callback - ERROR - Error processing callback for node 693ae9dafee8383e92b65e9e: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9dafee8383e92b65e9e (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9dafee8383e92b65e9e (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9dafee8383e92b65e9e (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:37:19,641 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e42fee8383e92b72591 2025-12-11 20:37:19,642 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e42fee8383e92b72591 HTTP/1.1" 502 150 2025-12-11 20:37:19,642 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e42fee8383e92b72591'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:37:20,497 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938887163d980a9748048be 2025-12-11 20:37:20,498 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938887163d980a9748048be HTTP/1.1" 502 150 2025-12-11 20:37:20,498 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938887163d980a9748048be'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:37:21,033 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b197d15f407b7629df654 2025-12-11 20:37:21,035 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b197d15f407b7629df654 HTTP/1.1" 502 150 2025-12-11 20:37:21,035 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b197d15f407b7629df654'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:37:23,642 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e42fee8383e92b72591 2025-12-11 20:37:23,643 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e42fee8383e92b72591 HTTP/1.1" 502 150 2025-12-11 20:37:23,643 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e42fee8383e92b72591'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:37:24,890 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0d68fee8383e92b721f9 2025-12-11 20:37:24,892 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d68fee8383e92b721f9 HTTP/1.1" 502 150 2025-12-11 20:37:24,892 - lava_callback - ERROR - Error processing callback for node 693b0d68fee8383e92b721f9: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0d68fee8383e92b721f9 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0d68fee8383e92b721f9 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0d68fee8383e92b721f9 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:51844 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:37:26,805 - lava_callback - INFO - Received callback for node 6938887863d980a9748048c4 INFO: 172.21.0.1:51846 - "POST /node/6938887863d980a9748048c4 HTTP/1.0" 202 Accepted 2025-12-11 20:37:26,811 - lava_callback - INFO - Starting async job submit for node 6938887863d980a9748048c4 2025-12-11 20:37:26,812 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:37:26,844 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938887863d980a9748048c4 HTTP/1.1" 502 150 2025-12-11 20:37:26,844 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938887863d980a9748048c4'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:37:26,844 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938887863d980a9748048c4 2025-12-11 20:37:26,845 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938887863d980a9748048c4 HTTP/1.1" 502 150 2025-12-11 20:37:26,845 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938887863d980a9748048c4'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:51856 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:37:28,845 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938887863d980a9748048c4 2025-12-11 20:37:28,846 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938887863d980a9748048c4 HTTP/1.1" 502 150 2025-12-11 20:37:28,847 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938887863d980a9748048c4'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:37:31,643 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e42fee8383e92b72591 2025-12-11 20:37:31,645 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e42fee8383e92b72591 HTTP/1.1" 502 150 2025-12-11 20:37:31,645 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e42fee8383e92b72591'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:37:32,847 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938887863d980a9748048c4 2025-12-11 20:37:32,848 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938887863d980a9748048c4 HTTP/1.1" 502 150 2025-12-11 20:37:32,848 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938887863d980a9748048c4'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:37:36,498 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938887163d980a9748048be 2025-12-11 20:37:36,504 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938887163d980a9748048be HTTP/1.1" 502 150 2025-12-11 20:37:36,504 - lava_callback - ERROR - Error processing callback for node 6938887163d980a9748048be: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938887163d980a9748048be (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938887163d980a9748048be (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938887163d980a9748048be (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:43388 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:37:37,035 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b197d15f407b7629df654 2025-12-11 20:37:37,036 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b197d15f407b7629df654 HTTP/1.1" 502 150 2025-12-11 20:37:37,036 - lava_callback - ERROR - Error processing callback for node 693b197d15f407b7629df654: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b197d15f407b7629df654 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b197d15f407b7629df654 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b197d15f407b7629df654 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:37:40,848 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938887863d980a9748048c4 2025-12-11 20:37:40,849 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938887863d980a9748048c4 HTTP/1.1" 502 150 2025-12-11 20:37:40,849 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938887863d980a9748048c4'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:39258 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:37:47,645 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e42fee8383e92b72591 2025-12-11 20:37:47,646 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e42fee8383e92b72591 HTTP/1.1" 502 150 2025-12-11 20:37:47,646 - lava_callback - ERROR - Error processing callback for node 693b0e42fee8383e92b72591: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e42fee8383e92b72591 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e42fee8383e92b72591 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e42fee8383e92b72591 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:39720 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:37:56,849 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938887863d980a9748048c4 2025-12-11 20:37:56,851 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938887863d980a9748048c4 HTTP/1.1" 502 150 2025-12-11 20:37:56,851 - lava_callback - ERROR - Error processing callback for node 6938887863d980a9748048c4: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938887863d980a9748048c4 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938887863d980a9748048c4 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938887863d980a9748048c4 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:38:04,262 - lava_callback - INFO - Received callback for node 693b0e43fee8383e92b72596 INFO: 172.21.0.1:60334 - "POST /node/693b0e43fee8383e92b72596 HTTP/1.0" 202 Accepted 2025-12-11 20:38:04,274 - lava_callback - INFO - Starting async job submit for node 693b0e43fee8383e92b72596 2025-12-11 20:38:04,324 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:38:04,507 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e43fee8383e92b72596 HTTP/1.1" 502 150 2025-12-11 20:38:04,507 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e43fee8383e92b72596'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:38:04,507 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e43fee8383e92b72596 2025-12-11 20:38:04,508 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e43fee8383e92b72596 HTTP/1.1" 502 150 2025-12-11 20:38:04,508 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e43fee8383e92b72596'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:38:06,509 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e43fee8383e92b72596 2025-12-11 20:38:06,510 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e43fee8383e92b72596 HTTP/1.1" 502 150 2025-12-11 20:38:06,510 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e43fee8383e92b72596'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:60346 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:38:07,006 - lava_callback - INFO - Received callback for node 693afae3fee8383e92b6d2f0 INFO: 172.21.0.1:60360 - "POST /node/693afae3fee8383e92b6d2f0 HTTP/1.0" 202 Accepted 2025-12-11 20:38:07,015 - lava_callback - INFO - Starting async job submit for node 693afae3fee8383e92b6d2f0 2025-12-11 20:38:07,048 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:38:07,064 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afae3fee8383e92b6d2f0 HTTP/1.1" 502 150 2025-12-11 20:38:07,064 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afae3fee8383e92b6d2f0'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:38:07,065 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afae3fee8383e92b6d2f0 2025-12-11 20:38:07,066 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afae3fee8383e92b6d2f0 HTTP/1.1" 502 150 2025-12-11 20:38:07,066 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afae3fee8383e92b6d2f0'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:38:09,066 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afae3fee8383e92b6d2f0 2025-12-11 20:38:09,067 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afae3fee8383e92b6d2f0 HTTP/1.1" 502 150 2025-12-11 20:38:09,068 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afae3fee8383e92b6d2f0'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:38:10,510 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e43fee8383e92b72596 2025-12-11 20:38:10,512 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e43fee8383e92b72596 HTTP/1.1" 502 150 2025-12-11 20:38:10,512 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e43fee8383e92b72596'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:38:13,068 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afae3fee8383e92b6d2f0 2025-12-11 20:38:13,069 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afae3fee8383e92b6d2f0 HTTP/1.1" 502 150 2025-12-11 20:38:13,069 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afae3fee8383e92b6d2f0'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:55356 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:55364 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:38:18,512 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e43fee8383e92b72596 2025-12-11 20:38:18,513 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e43fee8383e92b72596 HTTP/1.1" 502 150 2025-12-11 20:38:18,513 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0e43fee8383e92b72596'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:38:21,070 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afae3fee8383e92b6d2f0 2025-12-11 20:38:21,071 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afae3fee8383e92b6d2f0 HTTP/1.1" 502 150 2025-12-11 20:38:21,071 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afae3fee8383e92b6d2f0'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:38:21,248 - lava_callback - INFO - Received callback for node 6938899a63d980a9748048da INFO: 172.21.0.1:37994 - "POST /node/6938899a63d980a9748048da HTTP/1.0" 202 Accepted 2025-12-11 20:38:21,295 - lava_callback - INFO - Starting async job submit for node 6938899a63d980a9748048da 2025-12-11 20:38:21,328 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:38:21,339 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938899a63d980a9748048da HTTP/1.1" 502 150 2025-12-11 20:38:21,339 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938899a63d980a9748048da'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:38:21,339 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938899a63d980a9748048da 2025-12-11 20:38:21,340 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938899a63d980a9748048da HTTP/1.1" 502 150 2025-12-11 20:38:21,340 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938899a63d980a9748048da'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:38:23,341 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938899a63d980a9748048da 2025-12-11 20:38:23,342 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938899a63d980a9748048da HTTP/1.1" 502 150 2025-12-11 20:38:23,342 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938899a63d980a9748048da'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:37998 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:38:26,997 - lava_callback - INFO - Received callback for node 693b1ba215f407b7629e028b INFO: 172.21.0.1:38008 - "POST /node/693b1ba215f407b7629e028b HTTP/1.0" 202 Accepted 2025-12-11 20:38:27,003 - lava_callback - INFO - Starting async job submit for node 693b1ba215f407b7629e028b 2025-12-11 20:38:27,039 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:38:27,048 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ba215f407b7629e028b HTTP/1.1" 502 150 2025-12-11 20:38:27,048 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ba215f407b7629e028b'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:38:27,049 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ba215f407b7629e028b 2025-12-11 20:38:27,049 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ba215f407b7629e028b HTTP/1.1" 502 150 2025-12-11 20:38:27,049 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ba215f407b7629e028b'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:38:27,342 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938899a63d980a9748048da 2025-12-11 20:38:27,343 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938899a63d980a9748048da HTTP/1.1" 502 150 2025-12-11 20:38:27,343 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938899a63d980a9748048da'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:38016 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:38:29,050 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ba215f407b7629e028b 2025-12-11 20:38:29,051 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ba215f407b7629e028b HTTP/1.1" 502 150 2025-12-11 20:38:29,051 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ba215f407b7629e028b'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:38:33,056 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ba215f407b7629e028b 2025-12-11 20:38:33,057 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ba215f407b7629e028b HTTP/1.1" 502 150 2025-12-11 20:38:33,057 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ba215f407b7629e028b'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:38:34,514 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0e43fee8383e92b72596 2025-12-11 20:38:34,515 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0e43fee8383e92b72596 HTTP/1.1" 502 150 2025-12-11 20:38:34,516 - lava_callback - ERROR - Error processing callback for node 693b0e43fee8383e92b72596: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e43fee8383e92b72596 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e43fee8383e92b72596 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0e43fee8383e92b72596 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:38:35,343 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938899a63d980a9748048da 2025-12-11 20:38:35,345 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938899a63d980a9748048da HTTP/1.1" 502 150 2025-12-11 20:38:35,345 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938899a63d980a9748048da'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:52976 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:38:37,071 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afae3fee8383e92b6d2f0 2025-12-11 20:38:37,073 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afae3fee8383e92b6d2f0 HTTP/1.1" 502 150 2025-12-11 20:38:37,073 - lava_callback - ERROR - Error processing callback for node 693afae3fee8383e92b6d2f0: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afae3fee8383e92b6d2f0 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afae3fee8383e92b6d2f0 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afae3fee8383e92b6d2f0 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:38:41,057 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ba215f407b7629e028b 2025-12-11 20:38:41,059 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ba215f407b7629e028b HTTP/1.1" 502 150 2025-12-11 20:38:41,059 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ba215f407b7629e028b'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:38176 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:38:47,925 - lava_callback - INFO - Received callback for node 693ae9ebfee8383e92b65ed8 INFO: 172.21.0.1:38182 - "POST /node/693ae9ebfee8383e92b65ed8 HTTP/1.0" 202 Accepted 2025-12-11 20:38:47,932 - lava_callback - INFO - Starting async job submit for node 693ae9ebfee8383e92b65ed8 2025-12-11 20:38:48,119 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:38:48,158 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9ebfee8383e92b65ed8 HTTP/1.1" 502 150 2025-12-11 20:38:48,158 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9ebfee8383e92b65ed8'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:38:48,158 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9ebfee8383e92b65ed8 2025-12-11 20:38:48,159 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9ebfee8383e92b65ed8 HTTP/1.1" 502 150 2025-12-11 20:38:48,159 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9ebfee8383e92b65ed8'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:38:50,159 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9ebfee8383e92b65ed8 2025-12-11 20:38:50,161 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9ebfee8383e92b65ed8 HTTP/1.1" 502 150 2025-12-11 20:38:50,161 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9ebfee8383e92b65ed8'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:38:51,345 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938899a63d980a9748048da 2025-12-11 20:38:51,346 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938899a63d980a9748048da HTTP/1.1" 502 150 2025-12-11 20:38:51,346 - lava_callback - ERROR - Error processing callback for node 6938899a63d980a9748048da: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938899a63d980a9748048da (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938899a63d980a9748048da (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938899a63d980a9748048da (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:38:53,384 - lava_callback - INFO - Received callback for node 693b19ad15f407b7629df6b6 INFO: 172.21.0.1:50570 - "POST /node/693b19ad15f407b7629df6b6 HTTP/1.0" 202 Accepted 2025-12-11 20:38:53,399 - lava_callback - INFO - Starting async job submit for node 693b19ad15f407b7629df6b6 2025-12-11 20:38:53,438 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:38:53,469 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19ad15f407b7629df6b6 HTTP/1.1" 502 150 2025-12-11 20:38:53,469 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19ad15f407b7629df6b6'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:38:53,469 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19ad15f407b7629df6b6 2025-12-11 20:38:53,470 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19ad15f407b7629df6b6 HTTP/1.1" 502 150 2025-12-11 20:38:53,470 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19ad15f407b7629df6b6'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:38:53,531 - lava_callback - INFO - Received callback for node 693b0f17fee8383e92b72a44 INFO: 172.21.0.1:50586 - "POST /node/693b0f17fee8383e92b72a44 HTTP/1.0" 202 Accepted 2025-12-11 20:38:53,538 - lava_callback - INFO - Starting async job submit for node 693b0f17fee8383e92b72a44 2025-12-11 20:38:53,568 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:38:53,603 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f17fee8383e92b72a44 HTTP/1.1" 502 150 2025-12-11 20:38:53,603 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f17fee8383e92b72a44'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:38:53,603 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f17fee8383e92b72a44 2025-12-11 20:38:53,604 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f17fee8383e92b72a44 HTTP/1.1" 502 150 2025-12-11 20:38:53,604 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f17fee8383e92b72a44'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:38:54,161 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9ebfee8383e92b65ed8 2025-12-11 20:38:54,162 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9ebfee8383e92b65ed8 HTTP/1.1" 502 150 2025-12-11 20:38:54,162 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9ebfee8383e92b65ed8'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:38:55,470 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19ad15f407b7629df6b6 2025-12-11 20:38:55,472 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19ad15f407b7629df6b6 HTTP/1.1" 502 150 2025-12-11 20:38:55,472 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19ad15f407b7629df6b6'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:38:55,604 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f17fee8383e92b72a44 2025-12-11 20:38:55,606 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f17fee8383e92b72a44 HTTP/1.1" 502 150 2025-12-11 20:38:55,606 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f17fee8383e92b72a44'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:50602 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:38:57,059 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ba215f407b7629e028b 2025-12-11 20:38:57,060 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ba215f407b7629e028b HTTP/1.1" 502 150 2025-12-11 20:38:57,060 - lava_callback - ERROR - Error processing callback for node 693b1ba215f407b7629e028b: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1ba215f407b7629e028b (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1ba215f407b7629e028b (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1ba215f407b7629e028b (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:38:59,472 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19ad15f407b7629df6b6 2025-12-11 20:38:59,474 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19ad15f407b7629df6b6 HTTP/1.1" 502 150 2025-12-11 20:38:59,474 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19ad15f407b7629df6b6'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:38:59,606 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f17fee8383e92b72a44 2025-12-11 20:38:59,607 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f17fee8383e92b72a44 HTTP/1.1" 502 150 2025-12-11 20:38:59,607 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f17fee8383e92b72a44'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:39:02,162 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9ebfee8383e92b65ed8 2025-12-11 20:39:02,164 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9ebfee8383e92b65ed8 HTTP/1.1" 502 150 2025-12-11 20:39:02,164 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9ebfee8383e92b65ed8'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:55600 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:39:07,474 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19ad15f407b7629df6b6 2025-12-11 20:39:07,475 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19ad15f407b7629df6b6 HTTP/1.1" 502 150 2025-12-11 20:39:07,475 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19ad15f407b7629df6b6'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:39:07,607 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f17fee8383e92b72a44 2025-12-11 20:39:07,608 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f17fee8383e92b72a44 HTTP/1.1" 502 150 2025-12-11 20:39:07,608 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f17fee8383e92b72a44'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:47810 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47818 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:39:18,164 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9ebfee8383e92b65ed8 2025-12-11 20:39:18,165 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9ebfee8383e92b65ed8 HTTP/1.1" 502 150 2025-12-11 20:39:18,165 - lava_callback - ERROR - Error processing callback for node 693ae9ebfee8383e92b65ed8: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9ebfee8383e92b65ed8 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9ebfee8383e92b65ed8 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9ebfee8383e92b65ed8 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:39:23,476 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19ad15f407b7629df6b6 2025-12-11 20:39:23,477 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19ad15f407b7629df6b6 HTTP/1.1" 502 150 2025-12-11 20:39:23,477 - lava_callback - ERROR - Error processing callback for node 693b19ad15f407b7629df6b6: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b19ad15f407b7629df6b6 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b19ad15f407b7629df6b6 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b19ad15f407b7629df6b6 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:39:23,609 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f17fee8383e92b72a44 2025-12-11 20:39:23,610 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f17fee8383e92b72a44 HTTP/1.1" 502 150 2025-12-11 20:39:23,611 - lava_callback - ERROR - Error processing callback for node 693b0f17fee8383e92b72a44: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0f17fee8383e92b72a44 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0f17fee8383e92b72a44 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0f17fee8383e92b72a44 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:52876 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:39:26,843 - lava_callback - INFO - Received callback for node 693b0a65fee8383e92b71a4e INFO: 172.21.0.1:52884 - "POST /node/693b0a65fee8383e92b71a4e HTTP/1.0" 202 Accepted 2025-12-11 20:39:26,851 - lava_callback - INFO - Starting async job submit for node 693b0a65fee8383e92b71a4e 2025-12-11 20:39:26,913 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:39:26,946 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0a65fee8383e92b71a4e HTTP/1.1" 502 150 2025-12-11 20:39:26,947 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0a65fee8383e92b71a4e'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:39:26,947 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0a65fee8383e92b71a4e 2025-12-11 20:39:26,947 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0a65fee8383e92b71a4e HTTP/1.1" 502 150 2025-12-11 20:39:26,948 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0a65fee8383e92b71a4e'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:52898 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:39:28,948 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0a65fee8383e92b71a4e 2025-12-11 20:39:28,949 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0a65fee8383e92b71a4e HTTP/1.1" 502 150 2025-12-11 20:39:28,949 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0a65fee8383e92b71a4e'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:39:32,949 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0a65fee8383e92b71a4e 2025-12-11 20:39:32,950 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0a65fee8383e92b71a4e HTTP/1.1" 502 150 2025-12-11 20:39:32,950 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0a65fee8383e92b71a4e'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:53494 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:39:40,950 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0a65fee8383e92b71a4e 2025-12-11 20:39:40,952 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0a65fee8383e92b71a4e HTTP/1.1" 502 150 2025-12-11 20:39:40,952 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0a65fee8383e92b71a4e'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:52830 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:39:55,332 - lava_callback - INFO - Received callback for node 693b0f18fee8383e92b72a47 INFO: 172.21.0.1:57588 - "POST /node/693b0f18fee8383e92b72a47 HTTP/1.0" 202 Accepted 2025-12-11 20:39:55,340 - lava_callback - INFO - Starting async job submit for node 693b0f18fee8383e92b72a47 2025-12-11 20:39:55,378 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:39:55,410 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f18fee8383e92b72a47 HTTP/1.1" 502 150 2025-12-11 20:39:55,410 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f18fee8383e92b72a47'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:39:55,410 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f18fee8383e92b72a47 2025-12-11 20:39:55,411 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f18fee8383e92b72a47 HTTP/1.1" 502 150 2025-12-11 20:39:55,411 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f18fee8383e92b72a47'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:57604 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:39:56,952 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0a65fee8383e92b71a4e 2025-12-11 20:39:56,953 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0a65fee8383e92b71a4e HTTP/1.1" 502 150 2025-12-11 20:39:56,953 - lava_callback - ERROR - Error processing callback for node 693b0a65fee8383e92b71a4e: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0a65fee8383e92b71a4e (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0a65fee8383e92b71a4e (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0a65fee8383e92b71a4e (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:39:57,411 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f18fee8383e92b72a47 2025-12-11 20:39:57,413 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f18fee8383e92b72a47 HTTP/1.1" 502 150 2025-12-11 20:39:57,413 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f18fee8383e92b72a47'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:40:01,413 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f18fee8383e92b72a47 2025-12-11 20:40:01,414 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f18fee8383e92b72a47 HTTP/1.1" 502 150 2025-12-11 20:40:01,414 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f18fee8383e92b72a47'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:37000 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:40:09,414 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f18fee8383e92b72a47 2025-12-11 20:40:09,416 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f18fee8383e92b72a47 HTTP/1.1" 502 150 2025-12-11 20:40:09,416 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f18fee8383e92b72a47'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:41562 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:41574 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:40:25,416 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f18fee8383e92b72a47 2025-12-11 20:40:25,417 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f18fee8383e92b72a47 HTTP/1.1" 502 150 2025-12-11 20:40:25,417 - lava_callback - ERROR - Error processing callback for node 693b0f18fee8383e92b72a47: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0f18fee8383e92b72a47 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0f18fee8383e92b72a47 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0f18fee8383e92b72a47 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:33248 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33262 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:40:34,765 - lava_callback - INFO - Received callback for node 693b0cacfee8383e92b7206e INFO: 172.21.0.1:33798 - "POST /node/693b0cacfee8383e92b7206e HTTP/1.0" 202 Accepted 2025-12-11 20:40:34,774 - lava_callback - INFO - Starting async job submit for node 693b0cacfee8383e92b7206e 2025-12-11 20:40:34,837 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:40:34,855 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0cacfee8383e92b7206e HTTP/1.1" 502 150 2025-12-11 20:40:34,855 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0cacfee8383e92b7206e'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:40:34,855 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0cacfee8383e92b7206e 2025-12-11 20:40:34,856 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0cacfee8383e92b7206e HTTP/1.1" 502 150 2025-12-11 20:40:34,856 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0cacfee8383e92b7206e'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:33810 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:40:36,856 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0cacfee8383e92b7206e 2025-12-11 20:40:36,857 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0cacfee8383e92b7206e HTTP/1.1" 502 150 2025-12-11 20:40:36,857 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0cacfee8383e92b7206e'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:40:40,858 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0cacfee8383e92b7206e 2025-12-11 20:40:40,859 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0cacfee8383e92b7206e HTTP/1.1" 502 150 2025-12-11 20:40:40,859 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0cacfee8383e92b7206e'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:40:43,137 - lava_callback - INFO - Received callback for node 693b0f1ffee8383e92b72a61 INFO: 172.21.0.1:33060 - "POST /node/693b0f1ffee8383e92b72a61 HTTP/1.0" 202 Accepted 2025-12-11 20:40:43,144 - lava_callback - INFO - Starting async job submit for node 693b0f1ffee8383e92b72a61 2025-12-11 20:40:43,173 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:40:43,205 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f1ffee8383e92b72a61 HTTP/1.1" 502 150 2025-12-11 20:40:43,205 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f1ffee8383e92b72a61'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:40:43,205 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f1ffee8383e92b72a61 2025-12-11 20:40:43,206 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f1ffee8383e92b72a61 HTTP/1.1" 502 150 2025-12-11 20:40:43,206 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f1ffee8383e92b72a61'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:40:45,206 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f1ffee8383e92b72a61 2025-12-11 20:40:45,207 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f1ffee8383e92b72a61 HTTP/1.1" 502 150 2025-12-11 20:40:45,207 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f1ffee8383e92b72a61'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:33070 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:40:48,859 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0cacfee8383e92b7206e 2025-12-11 20:40:48,861 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0cacfee8383e92b7206e HTTP/1.1" 502 150 2025-12-11 20:40:48,861 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0cacfee8383e92b7206e'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:40:49,208 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f1ffee8383e92b72a61 2025-12-11 20:40:49,209 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f1ffee8383e92b72a61 HTTP/1.1" 502 150 2025-12-11 20:40:49,209 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f1ffee8383e92b72a61'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:40:51,972 - lava_callback - INFO - Received callback for node 693b02eefee8383e92b6f246 INFO: 172.21.0.1:34690 - "POST /node/693b02eefee8383e92b6f246 HTTP/1.0" 202 Accepted 2025-12-11 20:40:51,985 - lava_callback - INFO - Starting async job submit for node 693b02eefee8383e92b6f246 2025-12-11 20:40:52,084 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:40:52,107 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b02eefee8383e92b6f246 HTTP/1.1" 502 150 2025-12-11 20:40:52,107 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b02eefee8383e92b6f246'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:40:52,107 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b02eefee8383e92b6f246 2025-12-11 20:40:52,108 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b02eefee8383e92b6f246 HTTP/1.1" 502 150 2025-12-11 20:40:52,108 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b02eefee8383e92b6f246'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:40:54,109 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b02eefee8383e92b6f246 2025-12-11 20:40:54,110 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b02eefee8383e92b6f246 HTTP/1.1" 502 150 2025-12-11 20:40:54,110 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b02eefee8383e92b6f246'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:34698 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:40:57,209 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f1ffee8383e92b72a61 2025-12-11 20:40:57,211 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f1ffee8383e92b72a61 HTTP/1.1" 502 150 2025-12-11 20:40:57,211 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f1ffee8383e92b72a61'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:40:58,110 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b02eefee8383e92b6f246 2025-12-11 20:40:58,113 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b02eefee8383e92b6f246 HTTP/1.1" 502 150 2025-12-11 20:40:58,114 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b02eefee8383e92b6f246'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:41:04,861 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0cacfee8383e92b7206e 2025-12-11 20:41:04,862 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0cacfee8383e92b7206e HTTP/1.1" 502 150 2025-12-11 20:41:04,862 - lava_callback - ERROR - Error processing callback for node 693b0cacfee8383e92b7206e: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0cacfee8383e92b7206e (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0cacfee8383e92b7206e (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0cacfee8383e92b7206e (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:41:06,114 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b02eefee8383e92b6f246 2025-12-11 20:41:06,115 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b02eefee8383e92b6f246 HTTP/1.1" 502 150 2025-12-11 20:41:06,115 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b02eefee8383e92b6f246'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:59334 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:41:06,804 - lava_callback - INFO - Received callback for node 693afbe5fee8383e92b6d67a INFO: 172.21.0.1:59338 - "POST /node/693afbe5fee8383e92b6d67a HTTP/1.0" 202 Accepted 2025-12-11 20:41:06,810 - lava_callback - INFO - Starting async job submit for node 693afbe5fee8383e92b6d67a 2025-12-11 20:41:06,828 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:41:06,864 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afbe5fee8383e92b6d67a HTTP/1.1" 502 150 2025-12-11 20:41:06,864 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afbe5fee8383e92b6d67a'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:41:06,864 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afbe5fee8383e92b6d67a 2025-12-11 20:41:06,865 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afbe5fee8383e92b6d67a HTTP/1.1" 502 150 2025-12-11 20:41:06,865 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afbe5fee8383e92b6d67a'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:41:07,869 - lava_callback - INFO - Received callback for node 693ae9f0fee8383e92b65eeb INFO: 172.21.0.1:59340 - "POST /node/693ae9f0fee8383e92b65eeb HTTP/1.0" 202 Accepted 2025-12-11 20:41:07,875 - lava_callback - INFO - Starting async job submit for node 693ae9f0fee8383e92b65eeb 2025-12-11 20:41:07,909 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:41:07,944 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9f0fee8383e92b65eeb HTTP/1.1" 502 150 2025-12-11 20:41:07,944 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9f0fee8383e92b65eeb'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:41:07,944 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9f0fee8383e92b65eeb 2025-12-11 20:41:07,945 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9f0fee8383e92b65eeb HTTP/1.1" 502 150 2025-12-11 20:41:07,946 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9f0fee8383e92b65eeb'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:41:08,436 - lava_callback - INFO - Received callback for node 693b0f05fee8383e92b729d3 INFO: 172.21.0.1:59348 - "POST /node/693b0f05fee8383e92b729d3 HTTP/1.0" 202 Accepted 2025-12-11 20:41:08,442 - lava_callback - INFO - Starting async job submit for node 693b0f05fee8383e92b729d3 2025-12-11 20:41:08,477 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:41:08,509 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f05fee8383e92b729d3 HTTP/1.1" 502 150 2025-12-11 20:41:08,509 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f05fee8383e92b729d3'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:41:08,509 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f05fee8383e92b729d3 2025-12-11 20:41:08,510 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f05fee8383e92b729d3 HTTP/1.1" 502 150 2025-12-11 20:41:08,510 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f05fee8383e92b729d3'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:41:08,865 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afbe5fee8383e92b6d67a 2025-12-11 20:41:08,866 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afbe5fee8383e92b6d67a HTTP/1.1" 502 150 2025-12-11 20:41:08,867 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afbe5fee8383e92b6d67a'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:41:09,946 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9f0fee8383e92b65eeb 2025-12-11 20:41:09,947 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9f0fee8383e92b65eeb HTTP/1.1" 502 150 2025-12-11 20:41:09,947 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9f0fee8383e92b65eeb'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:41:10,510 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f05fee8383e92b729d3 2025-12-11 20:41:10,511 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f05fee8383e92b729d3 HTTP/1.1" 502 150 2025-12-11 20:41:10,511 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f05fee8383e92b729d3'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:41:12,867 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afbe5fee8383e92b6d67a 2025-12-11 20:41:12,868 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afbe5fee8383e92b6d67a HTTP/1.1" 502 150 2025-12-11 20:41:12,868 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afbe5fee8383e92b6d67a'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:41:13,211 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f1ffee8383e92b72a61 2025-12-11 20:41:13,212 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f1ffee8383e92b72a61 HTTP/1.1" 502 150 2025-12-11 20:41:13,212 - lava_callback - ERROR - Error processing callback for node 693b0f1ffee8383e92b72a61: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0f1ffee8383e92b72a61 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0f1ffee8383e92b72a61 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0f1ffee8383e92b72a61 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:41:13,947 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9f0fee8383e92b65eeb 2025-12-11 20:41:13,948 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9f0fee8383e92b65eeb HTTP/1.1" 502 150 2025-12-11 20:41:13,948 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9f0fee8383e92b65eeb'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:41:14,511 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f05fee8383e92b729d3 2025-12-11 20:41:14,513 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f05fee8383e92b729d3 HTTP/1.1" 502 150 2025-12-11 20:41:14,513 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f05fee8383e92b729d3'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:48266 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48282 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:41:20,868 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afbe5fee8383e92b6d67a 2025-12-11 20:41:20,869 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afbe5fee8383e92b6d67a HTTP/1.1" 502 150 2025-12-11 20:41:20,869 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afbe5fee8383e92b6d67a'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:41:21,949 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9f0fee8383e92b65eeb 2025-12-11 20:41:21,950 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9f0fee8383e92b65eeb HTTP/1.1" 502 150 2025-12-11 20:41:21,950 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9f0fee8383e92b65eeb'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:41:22,115 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b02eefee8383e92b6f246 2025-12-11 20:41:22,117 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b02eefee8383e92b6f246 HTTP/1.1" 502 150 2025-12-11 20:41:22,117 - lava_callback - ERROR - Error processing callback for node 693b02eefee8383e92b6f246: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b02eefee8383e92b6f246 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b02eefee8383e92b6f246 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b02eefee8383e92b6f246 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:41:22,513 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f05fee8383e92b729d3 2025-12-11 20:41:22,515 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f05fee8383e92b729d3 HTTP/1.1" 502 150 2025-12-11 20:41:22,515 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f05fee8383e92b729d3'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:52968 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52976 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54928 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:41:36,870 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afbe5fee8383e92b6d67a 2025-12-11 20:41:36,871 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afbe5fee8383e92b6d67a HTTP/1.1" 502 150 2025-12-11 20:41:36,871 - lava_callback - ERROR - Error processing callback for node 693afbe5fee8383e92b6d67a: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afbe5fee8383e92b6d67a (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afbe5fee8383e92b6d67a (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afbe5fee8383e92b6d67a (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:41:37,950 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9f0fee8383e92b65eeb 2025-12-11 20:41:37,951 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9f0fee8383e92b65eeb HTTP/1.1" 502 150 2025-12-11 20:41:37,951 - lava_callback - ERROR - Error processing callback for node 693ae9f0fee8383e92b65eeb: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9f0fee8383e92b65eeb (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9f0fee8383e92b65eeb (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9f0fee8383e92b65eeb (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:41:38,515 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f05fee8383e92b729d3 2025-12-11 20:41:38,516 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f05fee8383e92b729d3 HTTP/1.1" 502 150 2025-12-11 20:41:38,517 - lava_callback - ERROR - Error processing callback for node 693b0f05fee8383e92b729d3: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0f05fee8383e92b729d3 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0f05fee8383e92b729d3 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0f05fee8383e92b729d3 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:41:42,117 - lava_callback - INFO - Received callback for node 693b0aadfee8383e92b71af5 INFO: 172.21.0.1:40316 - "POST /node/693b0aadfee8383e92b71af5 HTTP/1.0" 202 Accepted 2025-12-11 20:41:42,130 - lava_callback - INFO - Starting async job submit for node 693b0aadfee8383e92b71af5 2025-12-11 20:41:42,217 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:41:42,250 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0aadfee8383e92b71af5 HTTP/1.1" 502 150 2025-12-11 20:41:42,250 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0aadfee8383e92b71af5'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:41:42,250 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0aadfee8383e92b71af5 2025-12-11 20:41:42,252 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0aadfee8383e92b71af5 HTTP/1.1" 502 150 2025-12-11 20:41:42,252 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0aadfee8383e92b71af5'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:41:44,252 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0aadfee8383e92b71af5 2025-12-11 20:41:44,253 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0aadfee8383e92b71af5 HTTP/1.1" 502 150 2025-12-11 20:41:44,253 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0aadfee8383e92b71af5'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:40328 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:41:48,253 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0aadfee8383e92b71af5 2025-12-11 20:41:48,255 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0aadfee8383e92b71af5 HTTP/1.1" 502 150 2025-12-11 20:41:48,255 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0aadfee8383e92b71af5'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:41:52,820 - lava_callback - INFO - Received callback for node 693b19ad15f407b7629df6b8 INFO: 172.21.0.1:54910 - "POST /node/693b19ad15f407b7629df6b8 HTTP/1.0" 202 Accepted 2025-12-11 20:41:52,828 - lava_callback - INFO - Starting async job submit for node 693b19ad15f407b7629df6b8 2025-12-11 20:41:52,862 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:41:52,874 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19ad15f407b7629df6b8 HTTP/1.1" 502 150 2025-12-11 20:41:52,874 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19ad15f407b7629df6b8'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:41:52,874 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19ad15f407b7629df6b8 2025-12-11 20:41:52,875 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19ad15f407b7629df6b8 HTTP/1.1" 502 150 2025-12-11 20:41:52,875 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19ad15f407b7629df6b8'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:41:54,875 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19ad15f407b7629df6b8 2025-12-11 20:41:54,876 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19ad15f407b7629df6b8 HTTP/1.1" 502 150 2025-12-11 20:41:54,876 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19ad15f407b7629df6b8'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:41:56,255 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0aadfee8383e92b71af5 2025-12-11 20:41:56,256 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0aadfee8383e92b71af5 HTTP/1.1" 502 150 2025-12-11 20:41:56,256 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0aadfee8383e92b71af5'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:54920 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:41:58,877 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19ad15f407b7629df6b8 2025-12-11 20:41:58,878 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19ad15f407b7629df6b8 HTTP/1.1" 502 150 2025-12-11 20:41:58,878 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19ad15f407b7629df6b8'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:50542 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:42:06,878 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19ad15f407b7629df6b8 2025-12-11 20:42:06,879 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19ad15f407b7629df6b8 HTTP/1.1" 502 150 2025-12-11 20:42:06,879 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19ad15f407b7629df6b8'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:08,185 - lava_callback - INFO - Received callback for node 693ae9f1fee8383e92b65ef2 INFO: 172.21.0.1:50556 - "POST /node/693ae9f1fee8383e92b65ef2 HTTP/1.0" 202 Accepted 2025-12-11 20:42:08,193 - lava_callback - INFO - Starting async job submit for node 693ae9f1fee8383e92b65ef2 2025-12-11 20:42:08,303 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:42:08,335 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9f1fee8383e92b65ef2 HTTP/1.1" 502 150 2025-12-11 20:42:08,335 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9f1fee8383e92b65ef2'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:08,335 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9f1fee8383e92b65ef2 2025-12-11 20:42:08,336 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9f1fee8383e92b65ef2 HTTP/1.1" 502 150 2025-12-11 20:42:08,336 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9f1fee8383e92b65ef2'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:09,883 - lava_callback - INFO - Received callback for node 6938899d63d980a9748048e1 INFO: 172.21.0.1:57048 - "POST /node/6938899d63d980a9748048e1 HTTP/1.0" 202 Accepted 2025-12-11 20:42:09,889 - lava_callback - INFO - Starting async job submit for node 6938899d63d980a9748048e1 2025-12-11 20:42:09,890 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:42:09,922 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938899d63d980a9748048e1 HTTP/1.1" 502 150 2025-12-11 20:42:09,922 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938899d63d980a9748048e1'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:09,922 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938899d63d980a9748048e1 2025-12-11 20:42:09,923 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938899d63d980a9748048e1 HTTP/1.1" 502 150 2025-12-11 20:42:09,923 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938899d63d980a9748048e1'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:10,336 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9f1fee8383e92b65ef2 2025-12-11 20:42:10,337 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9f1fee8383e92b65ef2 HTTP/1.1" 502 150 2025-12-11 20:42:10,337 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9f1fee8383e92b65ef2'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:10,556 - lava_callback - INFO - Received callback for node 6938899f63d980a9748048e4 INFO: 172.21.0.1:57050 - "POST /node/6938899f63d980a9748048e4 HTTP/1.0" 202 Accepted 2025-12-11 20:42:10,562 - lava_callback - INFO - Starting async job submit for node 6938899f63d980a9748048e4 2025-12-11 20:42:10,563 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:42:10,572 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938899f63d980a9748048e4 HTTP/1.1" 502 150 2025-12-11 20:42:10,572 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938899f63d980a9748048e4'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:10,572 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938899f63d980a9748048e4 2025-12-11 20:42:10,573 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938899f63d980a9748048e4 HTTP/1.1" 502 150 2025-12-11 20:42:10,573 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938899f63d980a9748048e4'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:11,923 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938899d63d980a9748048e1 2025-12-11 20:42:11,925 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938899d63d980a9748048e1 HTTP/1.1" 502 150 2025-12-11 20:42:11,925 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938899d63d980a9748048e1'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:12,257 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0aadfee8383e92b71af5 2025-12-11 20:42:12,258 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0aadfee8383e92b71af5 HTTP/1.1" 502 150 2025-12-11 20:42:12,258 - lava_callback - ERROR - Error processing callback for node 693b0aadfee8383e92b71af5: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0aadfee8383e92b71af5 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0aadfee8383e92b71af5 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0aadfee8383e92b71af5 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:42:12,573 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938899f63d980a9748048e4 2025-12-11 20:42:12,574 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938899f63d980a9748048e4 HTTP/1.1" 502 150 2025-12-11 20:42:12,574 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938899f63d980a9748048e4'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:14,338 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9f1fee8383e92b65ef2 2025-12-11 20:42:14,339 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9f1fee8383e92b65ef2 HTTP/1.1" 502 150 2025-12-11 20:42:14,339 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9f1fee8383e92b65ef2'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:57052 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 20:42:15,925 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938899d63d980a9748048e1 2025-12-11 20:42:15,926 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938899d63d980a9748048e1 HTTP/1.1" 502 150 2025-12-11 20:42:15,926 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938899d63d980a9748048e1'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:57060 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:42:16,575 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938899f63d980a9748048e4 2025-12-11 20:42:16,576 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938899f63d980a9748048e4 HTTP/1.1" 502 150 2025-12-11 20:42:16,576 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938899f63d980a9748048e4'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:22,339 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9f1fee8383e92b65ef2 2025-12-11 20:42:22,340 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9f1fee8383e92b65ef2 HTTP/1.1" 502 150 2025-12-11 20:42:22,340 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9f1fee8383e92b65ef2'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:22,879 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19ad15f407b7629df6b8 2025-12-11 20:42:22,881 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19ad15f407b7629df6b8 HTTP/1.1" 502 150 2025-12-11 20:42:22,881 - lava_callback - ERROR - Error processing callback for node 693b19ad15f407b7629df6b8: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b19ad15f407b7629df6b8 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b19ad15f407b7629df6b8 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b19ad15f407b7629df6b8 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:42:23,926 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938899d63d980a9748048e1 2025-12-11 20:42:23,928 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938899d63d980a9748048e1 HTTP/1.1" 502 150 2025-12-11 20:42:23,928 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938899d63d980a9748048e1'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:24,576 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938899f63d980a9748048e4 2025-12-11 20:42:24,577 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938899f63d980a9748048e4 HTTP/1.1" 502 150 2025-12-11 20:42:24,577 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/6938899f63d980a9748048e4'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:25,058 - lava_callback - INFO - Received callback for node 693894ea63d980a9748051bf INFO: 172.21.0.1:34664 - "POST /node/693894ea63d980a9748051bf HTTP/1.0" 202 Accepted 2025-12-11 20:42:25,065 - lava_callback - INFO - Starting async job submit for node 693894ea63d980a9748051bf 2025-12-11 20:42:25,095 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:42:25,114 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693894ea63d980a9748051bf HTTP/1.1" 502 150 2025-12-11 20:42:25,114 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693894ea63d980a9748051bf'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:25,114 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693894ea63d980a9748051bf 2025-12-11 20:42:25,115 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693894ea63d980a9748051bf HTTP/1.1" 502 150 2025-12-11 20:42:25,115 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693894ea63d980a9748051bf'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:34672 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:42:27,115 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693894ea63d980a9748051bf 2025-12-11 20:42:27,116 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693894ea63d980a9748051bf HTTP/1.1" 502 150 2025-12-11 20:42:27,116 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693894ea63d980a9748051bf'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:34674 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:42:29,876 - lava_callback - INFO - Received callback for node 693889a163d980a9748048e7 INFO: 172.21.0.1:53522 - "POST /node/693889a163d980a9748048e7 HTTP/1.0" 202 Accepted 2025-12-11 20:42:29,886 - lava_callback - INFO - Starting async job submit for node 693889a163d980a9748048e7 2025-12-11 20:42:29,887 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:42:29,921 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889a163d980a9748048e7 HTTP/1.1" 502 150 2025-12-11 20:42:29,921 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889a163d980a9748048e7'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:29,921 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889a163d980a9748048e7 2025-12-11 20:42:29,922 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889a163d980a9748048e7 HTTP/1.1" 502 150 2025-12-11 20:42:29,922 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889a163d980a9748048e7'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:30,557 - lava_callback - INFO - Received callback for node 693889a263d980a9748048ea INFO: 172.21.0.1:53526 - "POST /node/693889a263d980a9748048ea HTTP/1.0" 202 Accepted 2025-12-11 20:42:30,567 - lava_callback - INFO - Starting async job submit for node 693889a263d980a9748048ea 2025-12-11 20:42:30,568 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:42:30,581 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889a263d980a9748048ea HTTP/1.1" 502 150 2025-12-11 20:42:30,581 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889a263d980a9748048ea'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:30,582 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889a263d980a9748048ea 2025-12-11 20:42:30,583 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889a263d980a9748048ea HTTP/1.1" 502 150 2025-12-11 20:42:30,583 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889a263d980a9748048ea'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:31,117 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693894ea63d980a9748051bf 2025-12-11 20:42:31,118 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693894ea63d980a9748051bf HTTP/1.1" 502 150 2025-12-11 20:42:31,118 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693894ea63d980a9748051bf'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:31,250 - lava_callback - INFO - Received callback for node 693889a463d980a9748048ed INFO: 172.21.0.1:53542 - "POST /node/693889a463d980a9748048ed HTTP/1.0" 202 Accepted 2025-12-11 20:42:31,260 - lava_callback - INFO - Starting async job submit for node 693889a463d980a9748048ed 2025-12-11 20:42:31,261 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:42:31,291 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889a463d980a9748048ed HTTP/1.1" 502 150 2025-12-11 20:42:31,291 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889a463d980a9748048ed'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:31,292 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889a463d980a9748048ed 2025-12-11 20:42:31,292 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889a463d980a9748048ed HTTP/1.1" 502 150 2025-12-11 20:42:31,292 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889a463d980a9748048ed'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:31,922 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889a163d980a9748048e7 2025-12-11 20:42:31,923 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889a163d980a9748048e7 HTTP/1.1" 502 150 2025-12-11 20:42:31,923 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889a163d980a9748048e7'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:31,940 - lava_callback - INFO - Received callback for node 693889a663d980a9748048f0 INFO: 172.21.0.1:53550 - "POST /node/693889a663d980a9748048f0 HTTP/1.0" 202 Accepted 2025-12-11 20:42:31,945 - lava_callback - INFO - Starting async job submit for node 693889a663d980a9748048f0 2025-12-11 20:42:31,946 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:42:31,957 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889a663d980a9748048f0 HTTP/1.1" 502 150 2025-12-11 20:42:31,957 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889a663d980a9748048f0'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:31,957 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889a663d980a9748048f0 2025-12-11 20:42:31,958 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889a663d980a9748048f0 HTTP/1.1" 502 150 2025-12-11 20:42:31,958 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889a663d980a9748048f0'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:32,583 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889a263d980a9748048ea 2025-12-11 20:42:32,584 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889a263d980a9748048ea HTTP/1.1" 502 150 2025-12-11 20:42:32,584 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889a263d980a9748048ea'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:33,293 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889a463d980a9748048ed 2025-12-11 20:42:33,294 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889a463d980a9748048ed HTTP/1.1" 502 150 2025-12-11 20:42:33,294 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889a463d980a9748048ed'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:33,958 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889a663d980a9748048f0 2025-12-11 20:42:33,959 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889a663d980a9748048f0 HTTP/1.1" 502 150 2025-12-11 20:42:33,959 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889a663d980a9748048f0'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:35,924 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889a163d980a9748048e7 2025-12-11 20:42:35,925 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889a163d980a9748048e7 HTTP/1.1" 502 150 2025-12-11 20:42:35,925 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889a163d980a9748048e7'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:53566 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:42:36,585 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889a263d980a9748048ea 2025-12-11 20:42:36,586 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889a263d980a9748048ea HTTP/1.1" 502 150 2025-12-11 20:42:36,586 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889a263d980a9748048ea'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:37,295 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889a463d980a9748048ed 2025-12-11 20:42:37,296 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889a463d980a9748048ed HTTP/1.1" 502 150 2025-12-11 20:42:37,296 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889a463d980a9748048ed'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:37,960 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889a663d980a9748048f0 2025-12-11 20:42:37,961 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889a663d980a9748048f0 HTTP/1.1" 502 150 2025-12-11 20:42:37,961 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889a663d980a9748048f0'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:38,341 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9f1fee8383e92b65ef2 2025-12-11 20:42:38,342 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9f1fee8383e92b65ef2 HTTP/1.1" 502 150 2025-12-11 20:42:38,342 - lava_callback - ERROR - Error processing callback for node 693ae9f1fee8383e92b65ef2: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9f1fee8383e92b65ef2 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9f1fee8383e92b65ef2 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9f1fee8383e92b65ef2 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:42:39,119 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693894ea63d980a9748051bf 2025-12-11 20:42:39,120 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693894ea63d980a9748051bf HTTP/1.1" 502 150 2025-12-11 20:42:39,120 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693894ea63d980a9748051bf'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:39,929 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938899d63d980a9748048e1 2025-12-11 20:42:39,930 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938899d63d980a9748048e1 HTTP/1.1" 502 150 2025-12-11 20:42:39,930 - lava_callback - ERROR - Error processing callback for node 6938899d63d980a9748048e1: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938899d63d980a9748048e1 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938899d63d980a9748048e1 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938899d63d980a9748048e1 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:42:40,577 - urllib3.connectionpool - DEBUG - Retry: /latest/node/6938899f63d980a9748048e4 2025-12-11 20:42:40,584 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6938899f63d980a9748048e4 HTTP/1.1" 502 150 2025-12-11 20:42:40,585 - lava_callback - ERROR - Error processing callback for node 6938899f63d980a9748048e4: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938899f63d980a9748048e4 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938899f63d980a9748048e4 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/6938899f63d980a9748048e4 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:42:43,925 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889a163d980a9748048e7 2025-12-11 20:42:43,926 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889a163d980a9748048e7 HTTP/1.1" 502 150 2025-12-11 20:42:43,926 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889a163d980a9748048e7'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:44,586 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889a263d980a9748048ea 2025-12-11 20:42:44,588 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889a263d980a9748048ea HTTP/1.1" 502 150 2025-12-11 20:42:44,588 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889a263d980a9748048ea'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:45,296 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889a463d980a9748048ed 2025-12-11 20:42:45,297 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889a463d980a9748048ed HTTP/1.1" 502 150 2025-12-11 20:42:45,298 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889a463d980a9748048ed'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:42:45,961 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889a663d980a9748048f0 2025-12-11 20:42:45,962 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889a663d980a9748048f0 HTTP/1.1" 502 150 2025-12-11 20:42:45,962 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889a663d980a9748048f0'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:54678 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:42:55,120 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693894ea63d980a9748051bf 2025-12-11 20:42:55,127 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693894ea63d980a9748051bf HTTP/1.1" 502 150 2025-12-11 20:42:55,127 - lava_callback - ERROR - Error processing callback for node 693894ea63d980a9748051bf: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693894ea63d980a9748051bf (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693894ea63d980a9748051bf (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693894ea63d980a9748051bf (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:48576 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:42:59,927 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889a163d980a9748048e7 2025-12-11 20:42:59,928 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889a163d980a9748048e7 HTTP/1.1" 502 150 2025-12-11 20:42:59,928 - lava_callback - ERROR - Error processing callback for node 693889a163d980a9748048e7: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889a163d980a9748048e7 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889a163d980a9748048e7 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889a163d980a9748048e7 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:43:00,588 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889a263d980a9748048ea 2025-12-11 20:43:00,590 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889a263d980a9748048ea HTTP/1.1" 502 150 2025-12-11 20:43:00,590 - lava_callback - ERROR - Error processing callback for node 693889a263d980a9748048ea: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889a263d980a9748048ea (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889a263d980a9748048ea (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889a263d980a9748048ea (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:43:01,298 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889a463d980a9748048ed 2025-12-11 20:43:01,299 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889a463d980a9748048ed HTTP/1.1" 502 150 2025-12-11 20:43:01,299 - lava_callback - ERROR - Error processing callback for node 693889a463d980a9748048ed: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889a463d980a9748048ed (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889a463d980a9748048ed (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889a463d980a9748048ed (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:43:01,962 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889a663d980a9748048f0 2025-12-11 20:43:01,966 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889a663d980a9748048f0 HTTP/1.1" 502 150 2025-12-11 20:43:01,966 - lava_callback - ERROR - Error processing callback for node 693889a663d980a9748048f0: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889a663d980a9748048f0 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889a663d980a9748048f0 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889a663d980a9748048f0 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:35008 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:43:06,927 - lava_callback - INFO - Received callback for node 693b0f64fee8383e92b72b46 INFO: 172.21.0.1:35022 - "POST /node/693b0f64fee8383e92b72b46 HTTP/1.0" 202 Accepted 2025-12-11 20:43:06,940 - lava_callback - INFO - Starting async job submit for node 693b0f64fee8383e92b72b46 2025-12-11 20:43:06,978 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:43:06,997 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f64fee8383e92b72b46 HTTP/1.1" 502 150 2025-12-11 20:43:06,997 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f64fee8383e92b72b46'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:06,997 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f64fee8383e92b72b46 2025-12-11 20:43:06,998 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f64fee8383e92b72b46 HTTP/1.1" 502 150 2025-12-11 20:43:06,998 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f64fee8383e92b72b46'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:08,999 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f64fee8383e92b72b46 2025-12-11 20:43:09,001 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f64fee8383e92b72b46 HTTP/1.1" 502 150 2025-12-11 20:43:09,001 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f64fee8383e92b72b46'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:09,208 - lava_callback - INFO - Received callback for node 693889d763d980a974804903 INFO: 172.21.0.1:35036 - "POST /node/693889d763d980a974804903 HTTP/1.0" 202 Accepted 2025-12-11 20:43:09,217 - lava_callback - INFO - Starting async job submit for node 693889d763d980a974804903 2025-12-11 20:43:09,219 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:43:09,245 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889d763d980a974804903 HTTP/1.1" 502 150 2025-12-11 20:43:09,246 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889d763d980a974804903'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:09,246 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889d763d980a974804903 2025-12-11 20:43:09,247 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889d763d980a974804903 HTTP/1.1" 502 150 2025-12-11 20:43:09,247 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889d763d980a974804903'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:11,248 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889d763d980a974804903 2025-12-11 20:43:11,249 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889d763d980a974804903 HTTP/1.1" 502 150 2025-12-11 20:43:11,249 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889d763d980a974804903'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:13,001 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f64fee8383e92b72b46 2025-12-11 20:43:13,002 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f64fee8383e92b72b46 HTTP/1.1" 502 150 2025-12-11 20:43:13,002 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f64fee8383e92b72b46'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:14,760 - lava_callback - INFO - Received callback for node 693889e063d980a974804916 INFO: 172.21.0.1:47346 - "POST /node/693889e063d980a974804916 HTTP/1.0" 202 Accepted 2025-12-11 20:43:14,766 - lava_callback - INFO - Starting async job submit for node 693889e063d980a974804916 2025-12-11 20:43:14,767 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:43:14,779 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e063d980a974804916 HTTP/1.1" 502 150 2025-12-11 20:43:14,779 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889e063d980a974804916'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:14,779 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889e063d980a974804916 2025-12-11 20:43:14,780 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e063d980a974804916 HTTP/1.1" 502 150 2025-12-11 20:43:14,780 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889e063d980a974804916'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:47348 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 20:43:15,249 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889d763d980a974804903 2025-12-11 20:43:15,251 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889d763d980a974804903 HTTP/1.1" 502 150 2025-12-11 20:43:15,251 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889d763d980a974804903'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:47360 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:43:16,780 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889e063d980a974804916 2025-12-11 20:43:16,781 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e063d980a974804916 HTTP/1.1" 502 150 2025-12-11 20:43:16,782 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889e063d980a974804916'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:20,782 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889e063d980a974804916 2025-12-11 20:43:20,783 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e063d980a974804916 HTTP/1.1" 502 150 2025-12-11 20:43:20,783 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889e063d980a974804916'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:21,002 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f64fee8383e92b72b46 2025-12-11 20:43:21,003 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f64fee8383e92b72b46 HTTP/1.1" 502 150 2025-12-11 20:43:21,004 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f64fee8383e92b72b46'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:23,252 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889d763d980a974804903 2025-12-11 20:43:23,254 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889d763d980a974804903 HTTP/1.1" 502 150 2025-12-11 20:43:23,254 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889d763d980a974804903'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:59244 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59254 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:43:28,783 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889e063d980a974804916 2025-12-11 20:43:28,784 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e063d980a974804916 HTTP/1.1" 502 150 2025-12-11 20:43:28,785 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889e063d980a974804916'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:29,200 - lava_callback - INFO - Received callback for node 693889e963d980a974804922 INFO: 172.21.0.1:59268 - "POST /node/693889e963d980a974804922 HTTP/1.0" 202 Accepted 2025-12-11 20:43:29,206 - lava_callback - INFO - Starting async job submit for node 693889e963d980a974804922 2025-12-11 20:43:29,207 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:43:29,216 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e963d980a974804922 HTTP/1.1" 502 150 2025-12-11 20:43:29,216 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889e963d980a974804922'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:29,217 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889e963d980a974804922 2025-12-11 20:43:29,217 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e963d980a974804922 HTTP/1.1" 502 150 2025-12-11 20:43:29,217 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889e963d980a974804922'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:31,217 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889e963d980a974804922 2025-12-11 20:43:31,218 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e963d980a974804922 HTTP/1.1" 502 150 2025-12-11 20:43:31,219 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889e963d980a974804922'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:34,695 - lava_callback - INFO - Received callback for node 693889e363d980a974804919 INFO: 172.21.0.1:44752 - "POST /node/693889e363d980a974804919 HTTP/1.0" 202 Accepted 2025-12-11 20:43:34,700 - lava_callback - INFO - Starting async job submit for node 693889e363d980a974804919 2025-12-11 20:43:34,701 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:43:34,732 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e363d980a974804919 HTTP/1.1" 502 150 2025-12-11 20:43:34,732 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889e363d980a974804919'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:34,732 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889e363d980a974804919 2025-12-11 20:43:34,733 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e363d980a974804919 HTTP/1.1" 502 150 2025-12-11 20:43:34,733 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889e363d980a974804919'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:35,219 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889e963d980a974804922 2025-12-11 20:43:35,220 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e963d980a974804922 HTTP/1.1" 502 150 2025-12-11 20:43:35,220 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889e963d980a974804922'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:35,391 - lava_callback - INFO - Received callback for node 693889e463d980a97480491c INFO: 172.21.0.1:44756 - "POST /node/693889e463d980a97480491c HTTP/1.0" 202 Accepted 2025-12-11 20:43:35,398 - lava_callback - INFO - Starting async job submit for node 693889e463d980a97480491c 2025-12-11 20:43:35,398 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:43:35,413 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e463d980a97480491c HTTP/1.1" 502 150 2025-12-11 20:43:35,413 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889e463d980a97480491c'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:35,413 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889e463d980a97480491c 2025-12-11 20:43:35,414 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e463d980a97480491c HTTP/1.1" 502 150 2025-12-11 20:43:35,414 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889e463d980a97480491c'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:36,083 - lava_callback - INFO - Received callback for node 693889e663d980a97480491f INFO: 172.21.0.1:44770 - "POST /node/693889e663d980a97480491f HTTP/1.0" 202 Accepted 2025-12-11 20:43:36,109 - lava_callback - INFO - Starting async job submit for node 693889e663d980a97480491f 2025-12-11 20:43:36,110 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:43:36,143 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e663d980a97480491f HTTP/1.1" 502 150 2025-12-11 20:43:36,143 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889e663d980a97480491f'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:36,143 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889e663d980a97480491f 2025-12-11 20:43:36,144 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e663d980a97480491f HTTP/1.1" 502 150 2025-12-11 20:43:36,144 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889e663d980a97480491f'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:44782 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:43:36,733 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889e363d980a974804919 2025-12-11 20:43:36,734 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e363d980a974804919 HTTP/1.1" 502 150 2025-12-11 20:43:36,734 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889e363d980a974804919'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:37,004 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f64fee8383e92b72b46 2025-12-11 20:43:37,005 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f64fee8383e92b72b46 HTTP/1.1" 502 150 2025-12-11 20:43:37,005 - lava_callback - ERROR - Error processing callback for node 693b0f64fee8383e92b72b46: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0f64fee8383e92b72b46 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0f64fee8383e92b72b46 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0f64fee8383e92b72b46 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:43:37,414 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889e463d980a97480491c 2025-12-11 20:43:37,415 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e463d980a97480491c HTTP/1.1" 502 150 2025-12-11 20:43:37,415 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889e463d980a97480491c'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:38,144 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889e663d980a97480491f 2025-12-11 20:43:38,147 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e663d980a97480491f HTTP/1.1" 502 150 2025-12-11 20:43:38,147 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889e663d980a97480491f'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:39,255 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889d763d980a974804903 2025-12-11 20:43:39,257 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889d763d980a974804903 HTTP/1.1" 502 150 2025-12-11 20:43:39,257 - lava_callback - ERROR - Error processing callback for node 693889d763d980a974804903: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889d763d980a974804903 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889d763d980a974804903 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889d763d980a974804903 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:43:40,734 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889e363d980a974804919 2025-12-11 20:43:40,736 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e363d980a974804919 HTTP/1.1" 502 150 2025-12-11 20:43:40,736 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889e363d980a974804919'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:41,415 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889e463d980a97480491c 2025-12-11 20:43:41,417 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e463d980a97480491c HTTP/1.1" 502 150 2025-12-11 20:43:41,417 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889e463d980a97480491c'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:42,147 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889e663d980a97480491f 2025-12-11 20:43:42,148 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e663d980a97480491f HTTP/1.1" 502 150 2025-12-11 20:43:42,148 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889e663d980a97480491f'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:43,220 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889e963d980a974804922 2025-12-11 20:43:43,221 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e963d980a974804922 HTTP/1.1" 502 150 2025-12-11 20:43:43,221 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889e963d980a974804922'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:44,785 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889e063d980a974804916 2025-12-11 20:43:44,786 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e063d980a974804916 HTTP/1.1" 502 150 2025-12-11 20:43:44,786 - lava_callback - ERROR - Error processing callback for node 693889e063d980a974804916: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889e063d980a974804916 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889e063d980a974804916 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889e063d980a974804916 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:43:46,292 - lava_callback - INFO - Received callback for node 693aec95fee8383e92b66af4 INFO: 172.21.0.1:51736 - "POST /node/693aec95fee8383e92b66af4 HTTP/1.0" 202 Accepted 2025-12-11 20:43:46,341 - lava_callback - INFO - Starting async job submit for node 693aec95fee8383e92b66af4 2025-12-11 20:43:46,365 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:43:46,375 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aec95fee8383e92b66af4 HTTP/1.1" 502 150 2025-12-11 20:43:46,375 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aec95fee8383e92b66af4'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:46,375 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aec95fee8383e92b66af4 2025-12-11 20:43:46,376 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aec95fee8383e92b66af4 HTTP/1.1" 502 150 2025-12-11 20:43:46,376 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aec95fee8383e92b66af4'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:51742 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:43:46,789 - lava_callback - INFO - Received callback for node 693b1ad515f407b7629dff50 INFO: 172.21.0.1:51752 - "POST /node/693b1ad515f407b7629dff50 HTTP/1.0" 202 Accepted 2025-12-11 20:43:46,838 - lava_callback - INFO - Starting async job submit for node 693b1ad515f407b7629dff50 2025-12-11 20:43:46,888 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:43:46,897 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ad515f407b7629dff50 HTTP/1.1" 502 150 2025-12-11 20:43:46,897 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ad515f407b7629dff50'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:46,897 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ad515f407b7629dff50 2025-12-11 20:43:46,898 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ad515f407b7629dff50 HTTP/1.1" 502 150 2025-12-11 20:43:46,898 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ad515f407b7629dff50'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:48,080 - lava_callback - INFO - Received callback for node 693ae9f2fee8383e92b65ef8 INFO: 172.21.0.1:51768 - "POST /node/693ae9f2fee8383e92b65ef8 HTTP/1.0" 202 Accepted 2025-12-11 20:43:48,092 - lava_callback - INFO - Starting async job submit for node 693ae9f2fee8383e92b65ef8 2025-12-11 20:43:48,136 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:43:48,149 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9f2fee8383e92b65ef8 HTTP/1.1" 502 150 2025-12-11 20:43:48,149 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9f2fee8383e92b65ef8'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:48,150 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9f2fee8383e92b65ef8 2025-12-11 20:43:48,150 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9f2fee8383e92b65ef8 HTTP/1.1" 502 150 2025-12-11 20:43:48,150 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9f2fee8383e92b65ef8'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:48,376 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aec95fee8383e92b66af4 2025-12-11 20:43:48,377 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aec95fee8383e92b66af4 HTTP/1.1" 502 150 2025-12-11 20:43:48,377 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aec95fee8383e92b66af4'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:48,536 - lava_callback - INFO - Received callback for node 693b1ba715f407b7629e0294 INFO: 172.21.0.1:51774 - "POST /node/693b1ba715f407b7629e0294 HTTP/1.0" 202 Accepted 2025-12-11 20:43:48,548 - lava_callback - INFO - Starting async job submit for node 693b1ba715f407b7629e0294 2025-12-11 20:43:48,605 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:43:48,616 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ba715f407b7629e0294 HTTP/1.1" 502 150 2025-12-11 20:43:48,616 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ba715f407b7629e0294'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:48,616 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ba715f407b7629e0294 2025-12-11 20:43:48,617 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ba715f407b7629e0294 HTTP/1.1" 502 150 2025-12-11 20:43:48,617 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ba715f407b7629e0294'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:48,737 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889e363d980a974804919 2025-12-11 20:43:48,742 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e363d980a974804919 HTTP/1.1" 502 150 2025-12-11 20:43:48,742 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889e363d980a974804919'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:48,898 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ad515f407b7629dff50 2025-12-11 20:43:48,900 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ad515f407b7629dff50 HTTP/1.1" 502 150 2025-12-11 20:43:48,900 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ad515f407b7629dff50'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:49,417 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889e463d980a97480491c 2025-12-11 20:43:49,418 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e463d980a97480491c HTTP/1.1" 502 150 2025-12-11 20:43:49,418 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889e463d980a97480491c'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:50,148 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889e663d980a97480491f 2025-12-11 20:43:50,150 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e663d980a97480491f HTTP/1.1" 502 150 2025-12-11 20:43:50,150 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693889e663d980a97480491f'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:50,151 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9f2fee8383e92b65ef8 2025-12-11 20:43:50,151 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9f2fee8383e92b65ef8 HTTP/1.1" 502 150 2025-12-11 20:43:50,151 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9f2fee8383e92b65ef8'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:50,184 - lava_callback - INFO - Received callback for node 693b0d4ffee8383e92b7219a 2025-12-11 20:43:50,193 - lava_callback - INFO - Starting async job submit for node 693b0d4ffee8383e92b7219a INFO: 172.21.0.1:36772 - "POST /node/693b0d4ffee8383e92b7219a HTTP/1.0" 202 Accepted 2025-12-11 20:43:50,254 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:43:50,265 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d4ffee8383e92b7219a HTTP/1.1" 502 150 2025-12-11 20:43:50,265 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0d4ffee8383e92b7219a'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:50,266 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0d4ffee8383e92b7219a 2025-12-11 20:43:50,266 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d4ffee8383e92b7219a HTTP/1.1" 502 150 2025-12-11 20:43:50,266 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0d4ffee8383e92b7219a'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:50,617 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ba715f407b7629e0294 2025-12-11 20:43:50,618 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ba715f407b7629e0294 HTTP/1.1" 502 150 2025-12-11 20:43:50,619 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ba715f407b7629e0294'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:52,266 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0d4ffee8383e92b7219a 2025-12-11 20:43:52,267 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d4ffee8383e92b7219a HTTP/1.1" 502 150 2025-12-11 20:43:52,268 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0d4ffee8383e92b7219a'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:52,377 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aec95fee8383e92b66af4 2025-12-11 20:43:52,378 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aec95fee8383e92b66af4 HTTP/1.1" 502 150 2025-12-11 20:43:52,379 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aec95fee8383e92b66af4'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:52,900 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ad515f407b7629dff50 2025-12-11 20:43:52,901 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ad515f407b7629dff50 HTTP/1.1" 502 150 2025-12-11 20:43:52,901 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ad515f407b7629dff50'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:54,152 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9f2fee8383e92b65ef8 2025-12-11 20:43:54,153 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9f2fee8383e92b65ef8 HTTP/1.1" 502 150 2025-12-11 20:43:54,153 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9f2fee8383e92b65ef8'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:54,619 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ba715f407b7629e0294 2025-12-11 20:43:54,620 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ba715f407b7629e0294 HTTP/1.1" 502 150 2025-12-11 20:43:54,620 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ba715f407b7629e0294'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:43:56,268 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0d4ffee8383e92b7219a 2025-12-11 20:43:56,269 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d4ffee8383e92b7219a HTTP/1.1" 502 150 2025-12-11 20:43:56,269 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0d4ffee8383e92b7219a'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:36778 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:43:59,222 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889e963d980a974804922 2025-12-11 20:43:59,223 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e963d980a974804922 HTTP/1.1" 502 150 2025-12-11 20:43:59,223 - lava_callback - ERROR - Error processing callback for node 693889e963d980a974804922: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889e963d980a974804922 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889e963d980a974804922 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889e963d980a974804922 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:44:00,379 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aec95fee8383e92b66af4 2025-12-11 20:44:00,380 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aec95fee8383e92b66af4 HTTP/1.1" 502 150 2025-12-11 20:44:00,380 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693aec95fee8383e92b66af4'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:00,902 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ad515f407b7629dff50 2025-12-11 20:44:00,903 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ad515f407b7629dff50 HTTP/1.1" 502 150 2025-12-11 20:44:00,903 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ad515f407b7629dff50'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:02,153 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9f2fee8383e92b65ef8 2025-12-11 20:44:02,155 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9f2fee8383e92b65ef8 HTTP/1.1" 502 150 2025-12-11 20:44:02,156 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae9f2fee8383e92b65ef8'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:02,620 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ba715f407b7629e0294 2025-12-11 20:44:02,621 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ba715f407b7629e0294 HTTP/1.1" 502 150 2025-12-11 20:44:02,621 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1ba715f407b7629e0294'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:04,269 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0d4ffee8383e92b7219a 2025-12-11 20:44:04,270 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d4ffee8383e92b7219a HTTP/1.1" 502 150 2025-12-11 20:44:04,271 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0d4ffee8383e92b7219a'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:04,742 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889e363d980a974804919 2025-12-11 20:44:04,744 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e363d980a974804919 HTTP/1.1" 502 150 2025-12-11 20:44:04,744 - lava_callback - ERROR - Error processing callback for node 693889e363d980a974804919: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889e363d980a974804919 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889e363d980a974804919 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889e363d980a974804919 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:44:05,418 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889e463d980a97480491c 2025-12-11 20:44:05,420 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e463d980a97480491c HTTP/1.1" 502 150 2025-12-11 20:44:05,420 - lava_callback - ERROR - Error processing callback for node 693889e463d980a97480491c: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889e463d980a97480491c (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889e463d980a97480491c (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889e463d980a97480491c (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:44:06,150 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693889e663d980a97480491f 2025-12-11 20:44:06,152 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693889e663d980a97480491f HTTP/1.1" 502 150 2025-12-11 20:44:06,152 - lava_callback - ERROR - Error processing callback for node 693889e663d980a97480491f: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889e663d980a97480491f (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889e663d980a97480491f (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693889e663d980a97480491f (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:42978 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:44:07,212 - lava_callback - INFO - Received callback for node 693afc02fee8383e92b6d721 INFO: 172.21.0.1:42980 - "POST /node/693afc02fee8383e92b6d721 HTTP/1.0" 202 Accepted 2025-12-11 20:44:07,221 - lava_callback - INFO - Starting async job submit for node 693afc02fee8383e92b6d721 2025-12-11 20:44:07,243 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:44:07,261 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afc02fee8383e92b6d721 HTTP/1.1" 502 150 2025-12-11 20:44:07,261 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afc02fee8383e92b6d721'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:07,261 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afc02fee8383e92b6d721 2025-12-11 20:44:07,262 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afc02fee8383e92b6d721 HTTP/1.1" 502 150 2025-12-11 20:44:07,262 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afc02fee8383e92b6d721'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:08,780 - lava_callback - INFO - Received callback for node 693b197d15f407b7629df656 INFO: 172.21.0.1:42996 - "POST /node/693b197d15f407b7629df656 HTTP/1.0" 202 Accepted 2025-12-11 20:44:08,790 - lava_callback - INFO - Starting async job submit for node 693b197d15f407b7629df656 2025-12-11 20:44:08,841 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:44:08,861 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b197d15f407b7629df656 HTTP/1.1" 502 150 2025-12-11 20:44:08,861 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b197d15f407b7629df656'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:08,861 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b197d15f407b7629df656 2025-12-11 20:44:08,861 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b197d15f407b7629df656 HTTP/1.1" 502 150 2025-12-11 20:44:08,861 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b197d15f407b7629df656'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:09,263 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afc02fee8383e92b6d721 2025-12-11 20:44:09,264 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afc02fee8383e92b6d721 HTTP/1.1" 502 150 2025-12-11 20:44:09,264 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afc02fee8383e92b6d721'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:10,862 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b197d15f407b7629df656 2025-12-11 20:44:10,863 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b197d15f407b7629df656 HTTP/1.1" 502 150 2025-12-11 20:44:10,863 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b197d15f407b7629df656'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:13,264 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afc02fee8383e92b6d721 2025-12-11 20:44:13,266 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afc02fee8383e92b6d721 HTTP/1.1" 502 150 2025-12-11 20:44:13,266 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afc02fee8383e92b6d721'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:14,863 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b197d15f407b7629df656 2025-12-11 20:44:14,864 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b197d15f407b7629df656 HTTP/1.1" 502 150 2025-12-11 20:44:14,864 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b197d15f407b7629df656'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:57628 - "GET /api/metrics HTTP/1.0" 200 OK 2025-12-11 20:44:16,380 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693aec95fee8383e92b66af4 2025-12-11 20:44:16,381 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693aec95fee8383e92b66af4 HTTP/1.1" 502 150 2025-12-11 20:44:16,381 - lava_callback - ERROR - Error processing callback for node 693aec95fee8383e92b66af4: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aec95fee8383e92b66af4 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aec95fee8383e92b66af4 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693aec95fee8383e92b66af4 (Caused by ResponseError('too many 502 error responses')) INFO: 172.21.0.1:57636 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:44:16,708 - lava_callback - INFO - Received callback for node 693b0f3efee8383e92b72aba INFO: 172.21.0.1:57638 - "POST /node/693b0f3efee8383e92b72aba HTTP/1.0" 202 Accepted 2025-12-11 20:44:16,717 - lava_callback - INFO - Starting async job submit for node 693b0f3efee8383e92b72aba 2025-12-11 20:44:16,759 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:44:16,769 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f3efee8383e92b72aba HTTP/1.1" 502 150 2025-12-11 20:44:16,769 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f3efee8383e92b72aba'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:16,769 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f3efee8383e92b72aba 2025-12-11 20:44:16,770 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f3efee8383e92b72aba HTTP/1.1" 502 150 2025-12-11 20:44:16,770 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f3efee8383e92b72aba'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:16,903 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ad515f407b7629dff50 2025-12-11 20:44:16,904 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ad515f407b7629dff50 HTTP/1.1" 502 150 2025-12-11 20:44:16,904 - lava_callback - ERROR - Error processing callback for node 693b1ad515f407b7629dff50: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1ad515f407b7629dff50 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1ad515f407b7629dff50 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1ad515f407b7629dff50 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:44:18,156 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae9f2fee8383e92b65ef8 2025-12-11 20:44:18,157 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae9f2fee8383e92b65ef8 HTTP/1.1" 502 150 2025-12-11 20:44:18,157 - lava_callback - ERROR - Error processing callback for node 693ae9f2fee8383e92b65ef8: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9f2fee8383e92b65ef8 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9f2fee8383e92b65ef8 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae9f2fee8383e92b65ef8 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:44:18,622 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1ba715f407b7629e0294 2025-12-11 20:44:18,623 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1ba715f407b7629e0294 HTTP/1.1" 502 150 2025-12-11 20:44:18,623 - lava_callback - ERROR - Error processing callback for node 693b1ba715f407b7629e0294: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1ba715f407b7629e0294 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1ba715f407b7629e0294 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b1ba715f407b7629e0294 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:44:18,770 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f3efee8383e92b72aba 2025-12-11 20:44:18,771 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f3efee8383e92b72aba HTTP/1.1" 502 150 2025-12-11 20:44:18,772 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f3efee8383e92b72aba'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:20,271 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0d4ffee8383e92b7219a 2025-12-11 20:44:20,272 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0d4ffee8383e92b7219a HTTP/1.1" 502 150 2025-12-11 20:44:20,272 - lava_callback - ERROR - Error processing callback for node 693b0d4ffee8383e92b7219a: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0d4ffee8383e92b7219a (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0d4ffee8383e92b7219a (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0d4ffee8383e92b7219a (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:44:21,266 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afc02fee8383e92b6d721 2025-12-11 20:44:21,267 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afc02fee8383e92b6d721 HTTP/1.1" 502 150 2025-12-11 20:44:21,268 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693afc02fee8383e92b6d721'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:22,772 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f3efee8383e92b72aba 2025-12-11 20:44:22,773 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f3efee8383e92b72aba HTTP/1.1" 502 150 2025-12-11 20:44:22,773 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f3efee8383e92b72aba'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:22,864 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b197d15f407b7629df656 2025-12-11 20:44:22,866 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b197d15f407b7629df656 HTTP/1.1" 502 150 2025-12-11 20:44:22,866 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b197d15f407b7629df656'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:41290 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:44:27,369 - lava_callback - INFO - Received callback for node 693af7a9fee8383e92b6c15e INFO: 172.21.0.1:41294 - "POST /node/693af7a9fee8383e92b6c15e HTTP/1.0" 202 Accepted 2025-12-11 20:44:27,376 - lava_callback - INFO - Starting async job submit for node 693af7a9fee8383e92b6c15e 2025-12-11 20:44:27,405 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:44:27,434 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af7a9fee8383e92b6c15e HTTP/1.1" 502 150 2025-12-11 20:44:27,434 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af7a9fee8383e92b6c15e'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:27,434 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af7a9fee8383e92b6c15e 2025-12-11 20:44:27,435 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af7a9fee8383e92b6c15e HTTP/1.1" 502 150 2025-12-11 20:44:27,435 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af7a9fee8383e92b6c15e'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:41304 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:44:29,435 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af7a9fee8383e92b6c15e 2025-12-11 20:44:29,436 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af7a9fee8383e92b6c15e HTTP/1.1" 502 150 2025-12-11 20:44:29,437 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af7a9fee8383e92b6c15e'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:30,773 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f3efee8383e92b72aba 2025-12-11 20:44:30,775 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f3efee8383e92b72aba HTTP/1.1" 502 150 2025-12-11 20:44:30,775 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b0f3efee8383e92b72aba'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:33,437 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af7a9fee8383e92b6c15e 2025-12-11 20:44:33,438 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af7a9fee8383e92b6c15e HTTP/1.1" 502 150 2025-12-11 20:44:33,438 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af7a9fee8383e92b6c15e'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:59038 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:44:37,268 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693afc02fee8383e92b6d721 2025-12-11 20:44:37,270 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693afc02fee8383e92b6d721 HTTP/1.1" 502 150 2025-12-11 20:44:37,270 - lava_callback - ERROR - Error processing callback for node 693afc02fee8383e92b6d721: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afc02fee8383e92b6d721 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afc02fee8383e92b6d721 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693afc02fee8383e92b6d721 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:44:38,866 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b197d15f407b7629df656 2025-12-11 20:44:38,871 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b197d15f407b7629df656 HTTP/1.1" 502 150 2025-12-11 20:44:38,871 - lava_callback - ERROR - Error processing callback for node 693b197d15f407b7629df656: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b197d15f407b7629df656 (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b197d15f407b7629df656 (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b197d15f407b7629df656 (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:44:41,439 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af7a9fee8383e92b6c15e 2025-12-11 20:44:41,440 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af7a9fee8383e92b6c15e HTTP/1.1" 502 150 2025-12-11 20:44:41,440 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693af7a9fee8383e92b6c15e'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:54278 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:44:46,775 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b0f3efee8383e92b72aba 2025-12-11 20:44:46,777 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b0f3efee8383e92b72aba HTTP/1.1" 502 150 2025-12-11 20:44:46,777 - lava_callback - ERROR - Error processing callback for node 693b0f3efee8383e92b72aba: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0f3efee8383e92b72aba (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0f3efee8383e92b72aba (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b0f3efee8383e92b72aba (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:44:47,853 - lava_callback - INFO - Received callback for node 693ae647fee8383e92b64c8a INFO: 172.21.0.1:54288 - "POST /node/693ae647fee8383e92b64c8a HTTP/1.0" 202 Accepted 2025-12-11 20:44:47,872 - lava_callback - INFO - Starting async job submit for node 693ae647fee8383e92b64c8a 2025-12-11 20:44:48,024 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:44:48,067 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae647fee8383e92b64c8a HTTP/1.1" 502 150 2025-12-11 20:44:48,067 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae647fee8383e92b64c8a'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:48,067 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae647fee8383e92b64c8a 2025-12-11 20:44:48,068 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae647fee8383e92b64c8a HTTP/1.1" 502 150 2025-12-11 20:44:48,068 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae647fee8383e92b64c8a'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:49,352 - lava_callback - INFO - Received callback for node 693b102afee8383e92b72e0b INFO: 172.21.0.1:54304 - "POST /node/693b102afee8383e92b72e0b HTTP/1.0" 202 Accepted 2025-12-11 20:44:49,363 - lava_callback - INFO - Starting async job submit for node 693b102afee8383e92b72e0b 2025-12-11 20:44:49,429 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:44:49,467 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b102afee8383e92b72e0b HTTP/1.1" 502 150 2025-12-11 20:44:49,467 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b102afee8383e92b72e0b'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:49,467 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b102afee8383e92b72e0b 2025-12-11 20:44:49,468 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b102afee8383e92b72e0b HTTP/1.1" 502 150 2025-12-11 20:44:49,468 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b102afee8383e92b72e0b'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:50,069 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae647fee8383e92b64c8a 2025-12-11 20:44:50,072 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae647fee8383e92b64c8a HTTP/1.1" 502 150 2025-12-11 20:44:50,072 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae647fee8383e92b64c8a'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:51,470 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b102afee8383e92b72e0b 2025-12-11 20:44:51,471 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b102afee8383e92b72e0b HTTP/1.1" 502 150 2025-12-11 20:44:51,471 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b102afee8383e92b72e0b'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:53,204 - lava_callback - INFO - Received callback for node 693b19ae15f407b7629df6bc INFO: 172.21.0.1:42914 - "POST /node/693b19ae15f407b7629df6bc HTTP/1.0" 202 Accepted 2025-12-11 20:44:53,215 - lava_callback - INFO - Starting async job submit for node 693b19ae15f407b7629df6bc 2025-12-11 20:44:53,247 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:44:53,265 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19ae15f407b7629df6bc HTTP/1.1" 502 150 2025-12-11 20:44:53,265 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19ae15f407b7629df6bc'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:53,265 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19ae15f407b7629df6bc 2025-12-11 20:44:53,266 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19ae15f407b7629df6bc HTTP/1.1" 502 150 2025-12-11 20:44:53,266 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19ae15f407b7629df6bc'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:54,072 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae647fee8383e92b64c8a 2025-12-11 20:44:54,073 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae647fee8383e92b64c8a HTTP/1.1" 502 150 2025-12-11 20:44:54,073 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae647fee8383e92b64c8a'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:55,266 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19ae15f407b7629df6bc 2025-12-11 20:44:55,267 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19ae15f407b7629df6bc HTTP/1.1" 502 150 2025-12-11 20:44:55,267 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19ae15f407b7629df6bc'): Retry(total=2, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:44:55,471 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b102afee8383e92b72e0b 2025-12-11 20:44:55,472 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b102afee8383e92b72e0b HTTP/1.1" 502 150 2025-12-11 20:44:55,473 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b102afee8383e92b72e0b'): Retry(total=1, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:42924 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:44:57,440 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693af7a9fee8383e92b6c15e 2025-12-11 20:44:57,446 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693af7a9fee8383e92b6c15e HTTP/1.1" 502 150 2025-12-11 20:44:57,446 - lava_callback - ERROR - Error processing callback for node 693af7a9fee8383e92b6c15e: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af7a9fee8383e92b6c15e (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af7a9fee8383e92b6c15e (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693af7a9fee8383e92b6c15e (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:44:59,268 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19ae15f407b7629df6bc 2025-12-11 20:44:59,270 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19ae15f407b7629df6bc HTTP/1.1" 502 150 2025-12-11 20:44:59,270 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19ae15f407b7629df6bc'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:45:02,074 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae647fee8383e92b64c8a 2025-12-11 20:45:02,075 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae647fee8383e92b64c8a HTTP/1.1" 502 150 2025-12-11 20:45:02,075 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693ae647fee8383e92b64c8a'): Retry(total=0, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:45:03,473 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b102afee8383e92b72e0b 2025-12-11 20:45:03,474 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b102afee8383e92b72e0b HTTP/1.1" 502 150 2025-12-11 20:45:03,474 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b102afee8383e92b72e0b'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:54874 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:45:07,270 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19ae15f407b7629df6bc 2025-12-11 20:45:07,271 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19ae15f407b7629df6bc HTTP/1.1" 502 150 2025-12-11 20:45:07,271 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b19ae15f407b7629df6bc'): Retry(total=0, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:47198 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47208 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:45:18,075 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693ae647fee8383e92b64c8a 2025-12-11 20:45:18,083 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693ae647fee8383e92b64c8a HTTP/1.1" 502 150 2025-12-11 20:45:18,083 - lava_callback - ERROR - Error processing callback for node 693ae647fee8383e92b64c8a: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae647fee8383e92b64c8a (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae647fee8383e92b64c8a (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693ae647fee8383e92b64c8a (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:45:19,474 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b102afee8383e92b72e0b 2025-12-11 20:45:19,475 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b102afee8383e92b72e0b HTTP/1.1" 502 150 2025-12-11 20:45:19,476 - lava_callback - ERROR - Error processing callback for node 693b102afee8383e92b72e0b: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b102afee8383e92b72e0b (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b102afee8383e92b72e0b (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b102afee8383e92b72e0b (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:45:23,271 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b19ae15f407b7629df6bc 2025-12-11 20:45:23,273 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b19ae15f407b7629df6bc HTTP/1.1" 502 150 2025-12-11 20:45:23,273 - lava_callback - ERROR - Error processing callback for node 693b19ae15f407b7629df6bc: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b19ae15f407b7629df6bc (Caused by ResponseError('too many 502 error responses')) urllib3.exceptions.ResponseError: too many 502 error responses The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 942, in urlopen return self.urlopen( ^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 932, in urlopen retries = retries.increment(method, url, response=response, _pool=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b19ae15f407b7629df6bc (Caused by ResponseError('too many 502 error responses')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 264, in async_job_submit job_node = api_helper.api.node.get(node_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/latest.py", line 108, in get return self._get(f'node/{node_id}').json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/api/__init__.py", line 80, in _get resp = session.get( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 691, in send raise RetryError(e, request=request) requests.exceptions.RetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/node/693b19ae15f407b7629df6bc (Caused by ResponseError('too many 502 error responses')) 2025-12-11 20:45:25,667 - lava_callback - INFO - Received callback for node 69388a5363d980a97480492d INFO: 172.21.0.1:37878 - "POST /node/69388a5363d980a97480492d HTTP/1.0" 202 Accepted 2025-12-11 20:45:25,677 - lava_callback - INFO - Starting async job submit for node 69388a5363d980a97480492d 2025-12-11 20:45:25,679 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:45:25,708 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69388a5363d980a97480492d HTTP/1.1" 502 150 2025-12-11 20:45:25,709 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/69388a5363d980a97480492d'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:45:25,709 - urllib3.connectionpool - DEBUG - Retry: /latest/node/69388a5363d980a97480492d 2025-12-11 20:45:25,710 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69388a5363d980a97480492d HTTP/1.1" 502 150 2025-12-11 20:45:25,710 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/69388a5363d980a97480492d'): Retry(total=3, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:37894 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:45:27,337 - lava_callback - INFO - Received callback for node 693b1bb515f407b7629e02ae INFO: 172.21.0.1:37906 - "POST /node/693b1bb515f407b7629e02ae HTTP/1.0" 202 Accepted 2025-12-11 20:45:27,346 - lava_callback - INFO - Starting async job submit for node 693b1bb515f407b7629e02ae 2025-12-11 20:45:27,382 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-12-11 20:45:27,395 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bb515f407b7629e02ae HTTP/1.1" 502 150 2025-12-11 20:45:27,395 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bb515f407b7629e02ae'): Retry(total=4, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:45:27,395 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bb515f407b7629e02ae 2025-12-11 20:45:27,396 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bb515f407b7629e02ae HTTP/1.1" 502 150 2025-12-11 20:45:27,396 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bb515f407b7629e02ae'): Retry(total=3, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:45:27,710 - urllib3.connectionpool - DEBUG - Retry: /latest/node/69388a5363d980a97480492d 2025-12-11 20:45:27,711 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69388a5363d980a97480492d HTTP/1.1" 502 150 2025-12-11 20:45:27,711 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/69388a5363d980a97480492d'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: 172.21.0.1:37920 - "GET / HTTP/1.0" 200 OK 2025-12-11 20:45:29,396 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bb515f407b7629e02ae 2025-12-11 20:45:29,397 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bb515f407b7629e02ae HTTP/1.1" 502 150 2025-12-11 20:45:29,397 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bb515f407b7629e02ae'): Retry(total=2, connect=None, read=None, redirect=None, status=None) INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [1] 2025-12-11 20:45:31,711 - urllib3.connectionpool - DEBUG - Retry: /latest/node/69388a5363d980a97480492d 2025-12-11 20:45:31,713 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69388a5363d980a97480492d HTTP/1.1" 502 150 2025-12-11 20:45:31,713 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/69388a5363d980a97480492d'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:45:33,398 - urllib3.connectionpool - DEBUG - Retry: /latest/node/693b1bb515f407b7629e02ae 2025-12-11 20:45:33,399 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/693b1bb515f407b7629e02ae HTTP/1.1" 502 150 2025-12-11 20:45:33,399 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/693b1bb515f407b7629e02ae'): Retry(total=1, connect=None, read=None, redirect=None, status=None) 2025-12-11 20:45:39,713 - urllib3.connectionpool - DEBUG - Retry: /latest/node/69388a5363d980a97480492d 2025-12-11 20:45:39,717 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69388a5363d980a97480492d HTTP/1.1" 502 150 2025-12-11 20:45:39,717 - urllib3.util.retry - DEBUG - Incremented Retry for (url='/latest/node/69388a5363d980a97480492d'): Retry(total=0, connect=None, read=None, redirect=None, status=None)