INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) INFO: 172.21.0.1:51460 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51466 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44626 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:44640 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33976 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58382 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60966 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55624 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43312 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43324 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:14:08,907 - lava_callback - INFO - Received callback for node 69c63c3bb98d4d3c02067da0 2026-03-29 08:14:08,918 - lava_callback - INFO - Starting async job submit for node 69c63c3bb98d4d3c02067da0 INFO: 172.21.0.1:43336 - "POST /node/69c63c3bb98d4d3c02067da0 HTTP/1.0" 202 Accepted 2026-03-29 08:14:08,920 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:14:08,955 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63c3bb98d4d3c02067da0 HTTP/1.1" 200 None 2026-03-29 08:14:08,957 - lava_callback - DEBUG - Processing node 69c63c3bb98d4d3c02067da0: baseline-arm64-baylibre 2026-03-29 08:14:08,958 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c63c3bb98d4d3c02067da0 2026-03-29 08:14:08,959 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:14:08,978 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63c3bb98d4d3c02067da0 HTTP/1.1" 200 None 2026-03-29 08:14:08,980 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:14:08,999 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c086b1ce0c9bc43bccb63a HTTP/1.1" 200 None 2026-03-29 08:14:09,001 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:14:09,041 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c63c3bb98d4d3c02067da0 HTTP/1.1" 200 None 2026-03-29 08:14:09,053 - lava_callback - ERROR - Error processing callback for node 69c63c3bb98d4d3c02067da0: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:36054 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:36066 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45104 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58620 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41940 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49014 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57500 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57504 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44614 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:44620 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51384 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:15:28,875 - lava_callback - INFO - Received callback for node 69c63c8db98d4d3c02067daa INFO: 172.21.0.1:51398 - "POST /node/69c63c8db98d4d3c02067daa HTTP/1.0" 202 Accepted 2026-03-29 08:15:28,882 - lava_callback - INFO - Starting async job submit for node 69c63c8db98d4d3c02067daa 2026-03-29 08:15:28,883 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:15:28,917 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63c8db98d4d3c02067daa HTTP/1.1" 200 None 2026-03-29 08:15:28,918 - lava_callback - DEBUG - Processing node 69c63c8db98d4d3c02067daa: baseline-arm64-baylibre 2026-03-29 08:15:28,918 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c63c8db98d4d3c02067daa 2026-03-29 08:15:28,919 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:15:28,941 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63c8db98d4d3c02067daa HTTP/1.1" 200 None 2026-03-29 08:15:28,943 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:15:28,965 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c086b1ce0c9bc43bccb63a HTTP/1.1" 200 None 2026-03-29 08:15:28,967 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:15:28,998 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c63c8db98d4d3c02067daa HTTP/1.1" 200 None 2026-03-29 08:15:29,008 - lava_callback - ERROR - Error processing callback for node 69c63c8db98d4d3c02067daa: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:38564 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46026 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51988 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48016 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48028 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51688 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:51690 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43636 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37616 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56098 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51648 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51632 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51648 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:17:09,095 - lava_callback - INFO - Received callback for node 69c63cf0b98d4d3c02067dd4 INFO: 172.21.0.1:51650 - "POST /node/69c63cf0b98d4d3c02067dd4 HTTP/1.0" 202 Accepted 2026-03-29 08:17:09,101 - lava_callback - INFO - Starting async job submit for node 69c63cf0b98d4d3c02067dd4 2026-03-29 08:17:09,102 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:17:09,126 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63cf0b98d4d3c02067dd4 HTTP/1.1" 200 None 2026-03-29 08:17:09,127 - lava_callback - DEBUG - Processing node 69c63cf0b98d4d3c02067dd4: baseline-arm64-baylibre 2026-03-29 08:17:09,127 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c63cf0b98d4d3c02067dd4 2026-03-29 08:17:09,128 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:17:09,148 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63cf0b98d4d3c02067dd4 HTTP/1.1" 200 None 2026-03-29 08:17:09,150 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:17:09,173 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c086b1ce0c9bc43bccb63a HTTP/1.1" 200 None 2026-03-29 08:17:09,176 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:17:09,215 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c63cf0b98d4d3c02067dd4 HTTP/1.1" 200 None 2026-03-29 08:17:09,216 - lava_callback - ERROR - Error processing callback for node 69c63cf0b98d4d3c02067dd4: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:57008 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:57024 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60294 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33684 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38072 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34500 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37238 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37250 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40222 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:40224 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47770 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:18:29,035 - lava_callback - INFO - Received callback for node 69c63d41b98d4d3c02067e0a INFO: 172.21.0.1:47778 - "POST /node/69c63d41b98d4d3c02067e0a HTTP/1.0" 202 Accepted 2026-03-29 08:18:29,041 - lava_callback - INFO - Starting async job submit for node 69c63d41b98d4d3c02067e0a 2026-03-29 08:18:29,042 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:18:29,056 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63d41b98d4d3c02067e0a HTTP/1.1" 200 None 2026-03-29 08:18:29,057 - lava_callback - DEBUG - Processing node 69c63d41b98d4d3c02067e0a: baseline-arm64-baylibre 2026-03-29 08:18:29,057 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c63d41b98d4d3c02067e0a 2026-03-29 08:18:29,058 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:18:29,072 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63d41b98d4d3c02067e0a HTTP/1.1" 200 None 2026-03-29 08:18:29,073 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:18:29,087 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c086b1ce0c9bc43bccb63a HTTP/1.1" 200 None 2026-03-29 08:18:29,088 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:18:29,108 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c63d41b98d4d3c02067e0a HTTP/1.1" 200 None 2026-03-29 08:18:29,110 - lava_callback - ERROR - Error processing callback for node 69c63d41b98d4d3c02067e0a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:53968 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47888 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33852 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58892 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58908 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45936 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:45944 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55014 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50536 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51426 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50092 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40174 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40184 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48150 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48164 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:20:22,742 - lava_callback - INFO - Received callback for node 69c63db2b98d4d3c02067e19 INFO: 172.21.0.1:57298 - "POST /node/69c63db2b98d4d3c02067e19 HTTP/1.0" 202 Accepted 2026-03-29 08:20:22,748 - lava_callback - INFO - Starting async job submit for node 69c63db2b98d4d3c02067e19 2026-03-29 08:20:22,749 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:20:22,766 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63db2b98d4d3c02067e19 HTTP/1.1" 200 None 2026-03-29 08:20:22,767 - lava_callback - DEBUG - Processing node 69c63db2b98d4d3c02067e19: baseline-arm64-clabbe 2026-03-29 08:20:22,767 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c63db2b98d4d3c02067e19 2026-03-29 08:20:22,768 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:20:22,780 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63db2b98d4d3c02067e19 HTTP/1.1" 200 None 2026-03-29 08:20:22,782 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:20:22,794 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c086b1ce0c9bc43bccb63a HTTP/1.1" 200 None 2026-03-29 08:20:22,796 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:20:22,817 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c63db2b98d4d3c02067e19 HTTP/1.1" 200 None 2026-03-29 08:20:22,826 - lava_callback - ERROR - Error processing callback for node 69c63db2b98d4d3c02067e19: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:57300 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39412 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32840 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40134 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43870 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43878 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52358 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:52364 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51424 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:21:29,314 - lava_callback - INFO - Received callback for node 69c63dfab98d4d3c02067e29 INFO: 172.21.0.1:51428 - "POST /node/69c63dfab98d4d3c02067e29 HTTP/1.0" 202 Accepted 2026-03-29 08:21:29,320 - lava_callback - INFO - Starting async job submit for node 69c63dfab98d4d3c02067e29 2026-03-29 08:21:29,321 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:21:29,335 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63dfab98d4d3c02067e29 HTTP/1.1" 200 None 2026-03-29 08:21:29,336 - lava_callback - DEBUG - Processing node 69c63dfab98d4d3c02067e29: baseline-arm64-baylibre 2026-03-29 08:21:29,336 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c63dfab98d4d3c02067e29 2026-03-29 08:21:29,337 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:21:29,350 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63dfab98d4d3c02067e29 HTTP/1.1" 200 None 2026-03-29 08:21:29,351 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:21:29,364 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c086b1ce0c9bc43bccb63a HTTP/1.1" 200 None 2026-03-29 08:21:29,365 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:21:29,386 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c63dfab98d4d3c02067e29 HTTP/1.1" 200 None 2026-03-29 08:21:29,398 - lava_callback - ERROR - Error processing callback for node 69c63dfab98d4d3c02067e29: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:45748 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57690 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40828 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49026 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49038 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59998 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:60006 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47830 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35898 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55428 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:22:49,300 - lava_callback - INFO - Received callback for node 69c63e47b98d4d3c02067e30 INFO: 172.21.0.1:55440 - "POST /node/69c63e47b98d4d3c02067e30 HTTP/1.0" 202 Accepted 2026-03-29 08:22:49,306 - lava_callback - INFO - Starting async job submit for node 69c63e47b98d4d3c02067e30 2026-03-29 08:22:49,307 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:22:49,322 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63e47b98d4d3c02067e30 HTTP/1.1" 200 None 2026-03-29 08:22:49,323 - lava_callback - DEBUG - Processing node 69c63e47b98d4d3c02067e30: baseline-arm64-baylibre 2026-03-29 08:22:49,323 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c63e47b98d4d3c02067e30 2026-03-29 08:22:49,324 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:22:49,360 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63e47b98d4d3c02067e30 HTTP/1.1" 200 None 2026-03-29 08:22:49,362 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:22:49,375 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c086b1ce0c9bc43bccb63a HTTP/1.1" 200 None 2026-03-29 08:22:49,377 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:22:49,406 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c63e47b98d4d3c02067e30 HTTP/1.1" 200 None 2026-03-29 08:22:49,412 - lava_callback - ERROR - Error processing callback for node 69c63e47b98d4d3c02067e30: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:49712 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43226 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43234 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60634 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:60638 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60742 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51314 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41654 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49622 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:24:02,846 - lava_callback - INFO - Received callback for node 69c63e9db98d4d3c02067e43 INFO: 172.21.0.1:47226 - "POST /node/69c63e9db98d4d3c02067e43 HTTP/1.0" 202 Accepted 2026-03-29 08:24:02,852 - lava_callback - INFO - Starting async job submit for node 69c63e9db98d4d3c02067e43 2026-03-29 08:24:02,853 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:24:02,868 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63e9db98d4d3c02067e43 HTTP/1.1" 200 None 2026-03-29 08:24:02,869 - lava_callback - DEBUG - Processing node 69c63e9db98d4d3c02067e43: baseline-arm64-clabbe 2026-03-29 08:24:02,869 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c63e9db98d4d3c02067e43 2026-03-29 08:24:02,870 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:24:02,909 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63e9db98d4d3c02067e43 HTTP/1.1" 200 None 2026-03-29 08:24:02,910 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:24:02,956 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c086b1ce0c9bc43bccb63a HTTP/1.1" 200 None 2026-03-29 08:24:02,957 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:24:02,978 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c63e9db98d4d3c02067e43 HTTP/1.1" 200 None 2026-03-29 08:24:02,983 - lava_callback - ERROR - Error processing callback for node 69c63e9db98d4d3c02067e43: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:47234 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47250 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55058 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:55068 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42188 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52248 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41242 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59742 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47868 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47870 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52382 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:52386 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:25:22,899 - lava_callback - INFO - Received callback for node 69c63eedb98d4d3c02068405 INFO: 172.21.0.1:47396 - "POST /node/69c63eedb98d4d3c02068405 HTTP/1.0" 202 Accepted 2026-03-29 08:25:22,904 - lava_callback - INFO - Starting async job submit for node 69c63eedb98d4d3c02068405 2026-03-29 08:25:22,905 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:25:22,952 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63eedb98d4d3c02068405 HTTP/1.1" 200 None 2026-03-29 08:25:22,953 - lava_callback - DEBUG - Processing node 69c63eedb98d4d3c02068405: baseline-arm64-clabbe 2026-03-29 08:25:22,953 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c63eedb98d4d3c02068405 2026-03-29 08:25:22,953 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:25:22,967 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63eedb98d4d3c02068405 HTTP/1.1" 200 None 2026-03-29 08:25:22,968 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:25:22,981 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c086b1ce0c9bc43bccb63a HTTP/1.1" 200 None 2026-03-29 08:25:22,982 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:25:23,003 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c63eedb98d4d3c02068405 HTTP/1.1" 200 None 2026-03-29 08:25:23,008 - lava_callback - ERROR - Error processing callback for node 69c63eedb98d4d3c02068405: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:47408 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45656 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43654 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53004 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56310 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56312 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52228 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:52238 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55528 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38760 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60002 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:26:49,609 - lava_callback - INFO - Received callback for node 69c63f34b98d4d3c0206841c INFO: 172.21.0.1:60006 - "POST /node/69c63f34b98d4d3c0206841c HTTP/1.0" 202 Accepted 2026-03-29 08:26:49,615 - lava_callback - INFO - Starting async job submit for node 69c63f34b98d4d3c0206841c 2026-03-29 08:26:49,616 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:26:49,630 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63f34b98d4d3c0206841c HTTP/1.1" 200 None 2026-03-29 08:26:49,631 - lava_callback - DEBUG - Processing node 69c63f34b98d4d3c0206841c: baseline-arm64-baylibre 2026-03-29 08:26:49,631 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c63f34b98d4d3c0206841c 2026-03-29 08:26:49,632 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:26:49,654 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63f34b98d4d3c0206841c HTTP/1.1" 200 None 2026-03-29 08:26:49,656 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:26:49,669 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c086b1ce0c9bc43bccb63a HTTP/1.1" 200 None 2026-03-29 08:26:49,670 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:26:49,692 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c63f34b98d4d3c0206841c HTTP/1.1" 200 None 2026-03-29 08:26:49,700 - lava_callback - ERROR - Error processing callback for node 69c63f34b98d4d3c0206841c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:51460 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52738 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52754 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48200 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48204 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37598 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59618 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49004 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44028 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:28:03,029 - lava_callback - INFO - Received callback for node 69c63f89b98d4d3c0206842d INFO: 172.21.0.1:43552 - "POST /node/69c63f89b98d4d3c0206842d HTTP/1.0" 202 Accepted 2026-03-29 08:28:03,034 - lava_callback - INFO - Starting async job submit for node 69c63f89b98d4d3c0206842d 2026-03-29 08:28:03,035 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:28:03,050 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63f89b98d4d3c0206842d HTTP/1.1" 200 None 2026-03-29 08:28:03,051 - lava_callback - DEBUG - Processing node 69c63f89b98d4d3c0206842d: baseline-arm64-clabbe 2026-03-29 08:28:03,051 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c63f89b98d4d3c0206842d 2026-03-29 08:28:03,051 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:28:03,065 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63f89b98d4d3c0206842d HTTP/1.1" 200 None 2026-03-29 08:28:03,066 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:28:03,079 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c086b1ce0c9bc43bccb63a HTTP/1.1" 200 None 2026-03-29 08:28:03,081 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:28:03,105 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c63f89b98d4d3c0206842d HTTP/1.1" 200 None 2026-03-29 08:28:03,110 - lava_callback - ERROR - Error processing callback for node 69c63f89b98d4d3c0206842d: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:43564 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43572 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33292 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:33308 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53290 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60150 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49932 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41042 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36694 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36698 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59134 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:59148 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37672 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45140 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52544 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59796 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60686 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60694 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50798 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:50806 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45316 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:30:29,705 - lava_callback - INFO - Received callback for node 69c6401db98d4d3c0206844f INFO: 172.21.0.1:38370 - "POST /node/69c6401db98d4d3c0206844f HTTP/1.0" 202 Accepted 2026-03-29 08:30:29,711 - lava_callback - INFO - Starting async job submit for node 69c6401db98d4d3c0206844f 2026-03-29 08:30:29,711 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:30:29,747 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6401db98d4d3c0206844f HTTP/1.1" 200 None 2026-03-29 08:30:29,748 - lava_callback - DEBUG - Processing node 69c6401db98d4d3c0206844f: baseline-arm64-baylibre 2026-03-29 08:30:29,748 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c6401db98d4d3c0206844f 2026-03-29 08:30:29,749 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:30:29,774 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6401db98d4d3c0206844f HTTP/1.1" 200 None 2026-03-29 08:30:29,775 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:30:29,789 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c086b1ce0c9bc43bccb63a HTTP/1.1" 200 None 2026-03-29 08:30:29,791 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:30:29,812 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c6401db98d4d3c0206844f HTTP/1.1" 200 None 2026-03-29 08:30:29,813 - lava_callback - ERROR - Error processing callback for node 69c6401db98d4d3c0206844f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:38376 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49426 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53740 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44362 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44368 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58512 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:58514 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45582 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44924 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:31:40,166 - lava_callback - INFO - Received callback for node 69c8e2f06a4c026e16348346 INFO: 172.21.0.1:58204 - "POST /node/69c8e2f06a4c026e16348346 HTTP/1.0" 202 Accepted 2026-03-29 08:31:40,172 - lava_callback - INFO - Starting async job submit for node 69c8e2f06a4c026e16348346 2026-03-29 08:31:40,194 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:31:40,233 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e2f06a4c026e16348346 HTTP/1.1" 200 None 2026-03-29 08:31:40,233 - lava_callback - DEBUG - Processing node 69c8e2f06a4c026e16348346: baseline-arm 2026-03-29 08:31:40,419 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:31:40,641 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:31:40,642 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69c8e2f06a4c026e16348346/log.txt.gz 2026-03-29 08:31:40,750 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:31:40,989 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:31:40,991 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69c8e2f06a4c026e16348346/lava-logs.yaml 2026-03-29 08:31:40,996 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:31:41,152 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:31:41,154 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69c8e2f06a4c026e16348346/lava_callback.json.gz 2026-03-29 08:31:41,177 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:31:41,217 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e2f06a4c026e16348346 HTTP/1.1" 200 None 2026-03-29 08:31:41,218 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:31:41,232 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff06a4c026e16347d3c HTTP/1.1" 200 None 2026-03-29 08:31:41,233 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:31:41,259 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e2f06a4c026e16348346 HTTP/1.1" 200 None 2026-03-29 08:31:41,260 - lava_callback - INFO - Completed processing callback for node 69c8e2f06a4c026e16348346 INFO: 172.21.0.1:58218 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42692 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:32:03,227 - lava_callback - INFO - Received callback for node 69c64071b98d4d3c02068465 INFO: 172.21.0.1:42400 - "POST /node/69c64071b98d4d3c02068465 HTTP/1.0" 202 Accepted 2026-03-29 08:32:03,233 - lava_callback - INFO - Starting async job submit for node 69c64071b98d4d3c02068465 2026-03-29 08:32:03,234 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:32:03,253 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64071b98d4d3c02068465 HTTP/1.1" 200 None 2026-03-29 08:32:03,254 - lava_callback - DEBUG - Processing node 69c64071b98d4d3c02068465: baseline-arm64-clabbe 2026-03-29 08:32:03,254 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64071b98d4d3c02068465 2026-03-29 08:32:03,255 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:32:03,269 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64071b98d4d3c02068465 HTTP/1.1" 200 None 2026-03-29 08:32:03,270 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:32:03,283 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c086b1ce0c9bc43bccb63a HTTP/1.1" 200 None 2026-03-29 08:32:03,284 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:32:03,305 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64071b98d4d3c02068465 HTTP/1.1" 200 None 2026-03-29 08:32:03,307 - lava_callback - ERROR - Error processing callback for node 69c64071b98d4d3c02068465: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:42416 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42432 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:32:09,059 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:32:09,080 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:34776 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:34792 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53338 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33142 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50656 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60074 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36958 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36966 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36492 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:36502 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:33:23,350 - lava_callback - INFO - Received callback for node 69c640bfb98d4d3c02068473 INFO: 172.21.0.1:42042 - "POST /node/69c640bfb98d4d3c02068473 HTTP/1.0" 202 Accepted 2026-03-29 08:33:23,356 - lava_callback - INFO - Starting async job submit for node 69c640bfb98d4d3c02068473 2026-03-29 08:33:23,356 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:33:23,371 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c640bfb98d4d3c02068473 HTTP/1.1" 200 None 2026-03-29 08:33:23,372 - lava_callback - DEBUG - Processing node 69c640bfb98d4d3c02068473: baseline-arm64-clabbe 2026-03-29 08:33:23,372 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c640bfb98d4d3c02068473 2026-03-29 08:33:23,373 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:33:23,388 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c640bfb98d4d3c02068473 HTTP/1.1" 200 None 2026-03-29 08:33:23,390 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:33:23,404 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c086b1ce0c9bc43bccb63a HTTP/1.1" 200 None 2026-03-29 08:33:23,405 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:33:23,427 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c640bfb98d4d3c02068473 HTTP/1.1" 200 None 2026-03-29 08:33:23,428 - lava_callback - ERROR - Error processing callback for node 69c640bfb98d4d3c02068473: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:42052 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39288 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54302 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40686 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36146 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36154 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55110 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:55116 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44242 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:34:29,947 - lava_callback - INFO - Received callback for node 69c64105b98d4d3c020684b5 INFO: 172.21.0.1:60698 - "POST /node/69c64105b98d4d3c020684b5 HTTP/1.0" 202 Accepted 2026-03-29 08:34:29,953 - lava_callback - INFO - Starting async job submit for node 69c64105b98d4d3c020684b5 2026-03-29 08:34:29,954 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:34:29,969 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64105b98d4d3c020684b5 HTTP/1.1" 200 None 2026-03-29 08:34:29,970 - lava_callback - DEBUG - Processing node 69c64105b98d4d3c020684b5: baseline-arm64-baylibre 2026-03-29 08:34:29,971 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64105b98d4d3c020684b5 2026-03-29 08:34:29,971 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:34:29,989 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64105b98d4d3c020684b5 HTTP/1.1" 200 None 2026-03-29 08:34:29,991 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:34:30,014 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c086b1ce0c9bc43bccb63a HTTP/1.1" 200 None 2026-03-29 08:34:30,016 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:34:30,058 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64105b98d4d3c020684b5 HTTP/1.1" 200 None 2026-03-29 08:34:30,060 - lava_callback - ERROR - Error processing callback for node 69c64105b98d4d3c020684b5: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:60712 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55852 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39154 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34432 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34434 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53990 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:54006 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42334 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43122 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37616 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:35:50,014 - lava_callback - INFO - Received callback for node 69c64152b98d4d3c020684c7 INFO: 172.21.0.1:37166 - "POST /node/69c64152b98d4d3c020684c7 HTTP/1.0" 202 Accepted 2026-03-29 08:35:50,023 - lava_callback - INFO - Starting async job submit for node 69c64152b98d4d3c020684c7 2026-03-29 08:35:50,025 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:35:50,042 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64152b98d4d3c020684c7 HTTP/1.1" 200 None 2026-03-29 08:35:50,044 - lava_callback - DEBUG - Processing node 69c64152b98d4d3c020684c7: baseline-arm64-baylibre 2026-03-29 08:35:50,044 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64152b98d4d3c020684c7 2026-03-29 08:35:50,045 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:35:50,065 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64152b98d4d3c020684c7 HTTP/1.1" 200 None 2026-03-29 08:35:50,067 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:35:50,087 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c086b1ce0c9bc43bccb63a HTTP/1.1" 200 None 2026-03-29 08:35:50,089 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:35:50,132 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64152b98d4d3c020684c7 HTTP/1.1" 200 None 2026-03-29 08:35:50,134 - lava_callback - ERROR - Error processing callback for node 69c64152b98d4d3c020684c7: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:37172 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53518 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53528 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:36:12,355 - lava_callback - INFO - Received callback for node 69c8e4656a4c026e1634836d INFO: 172.21.0.1:48274 - "POST /node/69c8e4656a4c026e1634836d HTTP/1.0" 202 Accepted 2026-03-29 08:36:12,364 - lava_callback - INFO - Starting async job submit for node 69c8e4656a4c026e1634836d 2026-03-29 08:36:12,372 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:36:12,399 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4656a4c026e1634836d HTTP/1.1" 200 None 2026-03-29 08:36:12,401 - lava_callback - DEBUG - Processing node 69c8e4656a4c026e1634836d: baseline-x86 2026-03-29 08:36:12,410 - lava_callback - WARNING - Failed to upload log 2026-03-29 08:36:12,414 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:36:12,574 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:36:12,576 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69c8e4656a4c026e1634836d/lava-logs.yaml 2026-03-29 08:36:12,578 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:36:12,714 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:36:12,715 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69c8e4656a4c026e1634836d/lava_callback.json.gz 2026-03-29 08:36:12,719 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:36:12,748 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4656a4c026e1634836d HTTP/1.1" 200 None 2026-03-29 08:36:12,751 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:36:12,777 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e00a6a4c026e16347d61 HTTP/1.1" 200 None 2026-03-29 08:36:12,779 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:36:12,817 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4656a4c026e1634836d HTTP/1.1" 200 None 2026-03-29 08:36:12,829 - lava_callback - INFO - Completed processing callback for node 69c8e4656a4c026e1634836d INFO: 172.21.0.1:48276 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48286 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57904 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:36:28,010 - lava_callback - INFO - Received callback for node 69c8e4676a4c026e16348377 INFO: 172.21.0.1:57914 - "POST /node/69c8e4676a4c026e16348377 HTTP/1.0" 202 Accepted 2026-03-29 08:36:28,019 - lava_callback - INFO - Starting async job submit for node 69c8e4676a4c026e16348377 2026-03-29 08:36:28,059 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:36:28,097 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4676a4c026e16348377 HTTP/1.1" 200 None 2026-03-29 08:36:28,098 - lava_callback - DEBUG - Processing node 69c8e4676a4c026e16348377: baseline-x86 2026-03-29 08:36:28,357 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:36:28,491 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:36:28,493 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-x86-69c8e4676a4c026e16348377/log.txt.gz 2026-03-29 08:36:28,654 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:36:29,068 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:36:29,070 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69c8e4676a4c026e16348377/lava-logs.yaml 2026-03-29 08:36:29,075 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:36:29,209 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:36:29,211 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69c8e4676a4c026e16348377/lava_callback.json.gz 2026-03-29 08:36:29,230 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:36:29,249 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4676a4c026e16348377 HTTP/1.1" 200 None 2026-03-29 08:36:29,251 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:36:29,281 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e00a6a4c026e16347d61 HTTP/1.1" 200 None 2026-03-29 08:36:29,283 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:36:29,333 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4676a4c026e16348377 HTTP/1.1" 200 None 2026-03-29 08:36:29,334 - lava_callback - INFO - Completed processing callback for node 69c8e4676a4c026e16348377 INFO: 172.21.0.1:34660 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:36:39,083 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:36:39,106 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:41110 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:36:50,306 - lava_callback - INFO - Received callback for node 69c8e4966a4c026e163483a0 INFO: 172.21.0.1:48418 - "POST /node/69c8e4966a4c026e163483a0 HTTP/1.0" 202 Accepted 2026-03-29 08:36:50,312 - lava_callback - INFO - Starting async job submit for node 69c8e4966a4c026e163483a0 2026-03-29 08:36:50,315 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:36:50,330 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4966a4c026e163483a0 HTTP/1.1" 200 None 2026-03-29 08:36:50,331 - lava_callback - DEBUG - Processing node 69c8e4966a4c026e163483a0: baseline-arm 2026-03-29 08:36:50,336 - lava_callback - WARNING - Failed to upload log 2026-03-29 08:36:50,339 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:36:50,507 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:36:50,509 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69c8e4966a4c026e163483a0/lava-logs.yaml 2026-03-29 08:36:50,511 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:36:50,676 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:36:50,678 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69c8e4966a4c026e163483a0/lava_callback.json.gz 2026-03-29 08:36:50,681 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:36:50,695 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4966a4c026e163483a0 HTTP/1.1" 200 None 2026-03-29 08:36:50,697 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:36:50,710 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:36:50,711 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:36:50,731 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4966a4c026e163483a0 HTTP/1.1" 200 None 2026-03-29 08:36:50,743 - lava_callback - INFO - Completed processing callback for node 69c8e4966a4c026e163483a0 INFO: 172.21.0.1:48422 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:37:01,689 - lava_callback - INFO - Received callback for node 69c8e4666a4c026e16348371 INFO: 172.21.0.1:44212 - "POST /node/69c8e4666a4c026e16348371 HTTP/1.0" 202 Accepted 2026-03-29 08:37:01,695 - lava_callback - INFO - Starting async job submit for node 69c8e4666a4c026e16348371 2026-03-29 08:37:01,715 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:01,735 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4666a4c026e16348371 HTTP/1.1" 200 None 2026-03-29 08:37:01,736 - lava_callback - DEBUG - Processing node 69c8e4666a4c026e16348371: baseline-x86 2026-03-29 08:37:01,996 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:37:02,119 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:37:02,121 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-x86-69c8e4666a4c026e16348371/log.txt.gz 2026-03-29 08:37:02,303 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:37:02,503 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:37:02,505 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69c8e4666a4c026e16348371/lava-logs.yaml 2026-03-29 08:37:02,513 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:37:02,693 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:37:02,695 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69c8e4666a4c026e16348371/lava_callback.json.gz 2026-03-29 08:37:02,762 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:02,807 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4666a4c026e16348371 HTTP/1.1" 200 None 2026-03-29 08:37:02,810 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:02,834 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e00a6a4c026e16347d61 HTTP/1.1" 200 None 2026-03-29 08:37:02,835 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:02,866 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4666a4c026e16348371 HTTP/1.1" 200 None 2026-03-29 08:37:02,867 - lava_callback - INFO - Completed processing callback for node 69c8e4666a4c026e16348371 2026-03-29 08:37:03,568 - lava_callback - INFO - Received callback for node 69c641a5b98d4d3c02068512 INFO: 172.21.0.1:44216 - "POST /node/69c641a5b98d4d3c02068512 HTTP/1.0" 202 Accepted 2026-03-29 08:37:03,578 - lava_callback - INFO - Starting async job submit for node 69c641a5b98d4d3c02068512 2026-03-29 08:37:03,582 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 INFO: 172.21.0.1:44228 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:37:03,604 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c641a5b98d4d3c02068512 HTTP/1.1" 200 None 2026-03-29 08:37:03,606 - lava_callback - DEBUG - Processing node 69c641a5b98d4d3c02068512: baseline-arm64-clabbe 2026-03-29 08:37:03,606 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c641a5b98d4d3c02068512 2026-03-29 08:37:03,607 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:03,626 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c641a5b98d4d3c02068512 HTTP/1.1" 200 None 2026-03-29 08:37:03,628 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:03,642 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c086b1ce0c9bc43bccb63a HTTP/1.1" 200 None 2026-03-29 08:37:03,643 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:03,679 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c641a5b98d4d3c02068512 HTTP/1.1" 200 None 2026-03-29 08:37:03,681 - lava_callback - ERROR - Error processing callback for node 69c641a5b98d4d3c02068512: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:44232 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:37:09,109 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:09,136 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-29 08:37:10,096 - lava_callback - INFO - Received callback for node 69c641a2b98d4d3c0206850a INFO: 172.21.0.1:36694 - "POST /node/69c641a2b98d4d3c0206850a HTTP/1.0" 202 Accepted 2026-03-29 08:37:10,106 - lava_callback - INFO - Starting async job submit for node 69c641a2b98d4d3c0206850a 2026-03-29 08:37:10,108 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:10,151 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c641a2b98d4d3c0206850a HTTP/1.1" 200 None 2026-03-29 08:37:10,152 - lava_callback - DEBUG - Processing node 69c641a2b98d4d3c0206850a: baseline-arm-baylibre 2026-03-29 08:37:10,153 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c641a2b98d4d3c0206850a 2026-03-29 08:37:10,154 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:10,177 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c641a2b98d4d3c0206850a HTTP/1.1" 200 None 2026-03-29 08:37:10,179 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:10,232 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63ce4b98d4d3c02067dbf HTTP/1.1" 200 None 2026-03-29 08:37:10,235 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:10,281 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c641a2b98d4d3c0206850a HTTP/1.1" 200 None 2026-03-29 08:37:10,282 - lava_callback - ERROR - Error processing callback for node 69c641a2b98d4d3c0206850a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-29 08:37:10,761 - lava_callback - INFO - Received callback for node 69c641a3b98d4d3c0206850d INFO: 172.21.0.1:36696 - "POST /node/69c641a3b98d4d3c0206850d HTTP/1.0" 202 Accepted 2026-03-29 08:37:10,767 - lava_callback - INFO - Starting async job submit for node 69c641a3b98d4d3c0206850d 2026-03-29 08:37:10,768 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:10,783 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c641a3b98d4d3c0206850d HTTP/1.1" 200 None 2026-03-29 08:37:10,784 - lava_callback - DEBUG - Processing node 69c641a3b98d4d3c0206850d: baseline-arm-baylibre 2026-03-29 08:37:10,784 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c641a3b98d4d3c0206850d 2026-03-29 08:37:10,785 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:10,804 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c641a3b98d4d3c0206850d HTTP/1.1" 200 None 2026-03-29 08:37:10,805 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:10,824 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63ce4b98d4d3c02067dbf HTTP/1.1" 200 None 2026-03-29 08:37:10,826 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:10,860 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c641a3b98d4d3c0206850d HTTP/1.1" 200 None 2026-03-29 08:37:10,861 - lava_callback - ERROR - Error processing callback for node 69c641a3b98d4d3c0206850d: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-29 08:37:11,443 - lava_callback - INFO - Received callback for node 69c641a5b98d4d3c02068510 INFO: 172.21.0.1:36706 - "POST /node/69c641a5b98d4d3c02068510 HTTP/1.0" 202 Accepted 2026-03-29 08:37:11,449 - lava_callback - INFO - Starting async job submit for node 69c641a5b98d4d3c02068510 2026-03-29 08:37:11,450 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:11,474 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c641a5b98d4d3c02068510 HTTP/1.1" 200 None 2026-03-29 08:37:11,475 - lava_callback - DEBUG - Processing node 69c641a5b98d4d3c02068510: baseline-arm-baylibre 2026-03-29 08:37:11,476 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c641a5b98d4d3c02068510 2026-03-29 08:37:11,476 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:11,510 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c641a5b98d4d3c02068510 HTTP/1.1" 200 None 2026-03-29 08:37:11,511 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:11,526 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63ce4b98d4d3c02067dbf HTTP/1.1" 200 None 2026-03-29 08:37:11,528 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:11,548 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c641a5b98d4d3c02068510 HTTP/1.1" 200 None 2026-03-29 08:37:11,553 - lava_callback - ERROR - Error processing callback for node 69c641a5b98d4d3c02068510: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-29 08:37:12,135 - lava_callback - INFO - Received callback for node 69c641a6b98d4d3c02068515 INFO: 172.21.0.1:36722 - "POST /node/69c641a6b98d4d3c02068515 HTTP/1.0" 202 Accepted 2026-03-29 08:37:12,151 - lava_callback - INFO - Starting async job submit for node 69c641a6b98d4d3c02068515 2026-03-29 08:37:12,152 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:12,185 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c641a6b98d4d3c02068515 HTTP/1.1" 200 None 2026-03-29 08:37:12,187 - lava_callback - DEBUG - Processing node 69c641a6b98d4d3c02068515: baseline-arm-baylibre 2026-03-29 08:37:12,187 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c641a6b98d4d3c02068515 2026-03-29 08:37:12,188 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:12,209 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c641a6b98d4d3c02068515 HTTP/1.1" 200 None 2026-03-29 08:37:12,211 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:12,225 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63ce4b98d4d3c02067dbf HTTP/1.1" 200 None 2026-03-29 08:37:12,226 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:12,245 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c641a6b98d4d3c02068515 HTTP/1.1" 200 None 2026-03-29 08:37:12,247 - lava_callback - ERROR - Error processing callback for node 69c641a6b98d4d3c02068515: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-29 08:37:12,828 - lava_callback - INFO - Received callback for node 69c641a9b98d4d3c02068521 INFO: 172.21.0.1:36736 - "POST /node/69c641a9b98d4d3c02068521 HTTP/1.0" 202 Accepted 2026-03-29 08:37:12,834 - lava_callback - INFO - Starting async job submit for node 69c641a9b98d4d3c02068521 2026-03-29 08:37:12,835 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:12,850 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c641a9b98d4d3c02068521 HTTP/1.1" 200 None 2026-03-29 08:37:12,851 - lava_callback - DEBUG - Processing node 69c641a9b98d4d3c02068521: baseline-arm-baylibre 2026-03-29 08:37:12,851 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c641a9b98d4d3c02068521 2026-03-29 08:37:12,852 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:12,871 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c641a9b98d4d3c02068521 HTTP/1.1" 200 None 2026-03-29 08:37:12,872 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:12,886 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63ce4b98d4d3c02067dbf HTTP/1.1" 200 None 2026-03-29 08:37:12,887 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:12,908 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c641a9b98d4d3c02068521 HTTP/1.1" 200 None 2026-03-29 08:37:12,909 - lava_callback - ERROR - Error processing callback for node 69c641a9b98d4d3c02068521: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-29 08:37:13,486 - lava_callback - INFO - Received callback for node 69c641aab98d4d3c02068524 INFO: 172.21.0.1:36744 - "POST /node/69c641aab98d4d3c02068524 HTTP/1.0" 202 Accepted 2026-03-29 08:37:13,492 - lava_callback - INFO - Starting async job submit for node 69c641aab98d4d3c02068524 2026-03-29 08:37:13,492 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:13,507 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c641aab98d4d3c02068524 HTTP/1.1" 200 None 2026-03-29 08:37:13,508 - lava_callback - DEBUG - Processing node 69c641aab98d4d3c02068524: baseline-arm-baylibre 2026-03-29 08:37:13,508 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c641aab98d4d3c02068524 2026-03-29 08:37:13,509 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:13,522 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c641aab98d4d3c02068524 HTTP/1.1" 200 None 2026-03-29 08:37:13,523 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:13,537 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63ce4b98d4d3c02067dbf HTTP/1.1" 200 None 2026-03-29 08:37:13,538 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:13,570 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c641aab98d4d3c02068524 HTTP/1.1" 200 None 2026-03-29 08:37:13,576 - lava_callback - ERROR - Error processing callback for node 69c641aab98d4d3c02068524: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:36760 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:36766 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:37:22,016 - lava_callback - INFO - Received callback for node 69c8e4996a4c026e163483b0 INFO: 172.21.0.1:44608 - "POST /node/69c8e4996a4c026e163483b0 HTTP/1.0" 202 Accepted 2026-03-29 08:37:22,066 - lava_callback - INFO - Starting async job submit for node 69c8e4996a4c026e163483b0 2026-03-29 08:37:22,084 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:22,102 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4996a4c026e163483b0 HTTP/1.1" 200 None 2026-03-29 08:37:22,103 - lava_callback - DEBUG - Processing node 69c8e4996a4c026e163483b0: baseline-arm 2026-03-29 08:37:22,228 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:37:22,342 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:37:22,343 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69c8e4996a4c026e163483b0/log.txt.gz 2026-03-29 08:37:22,418 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:37:22,575 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:37:22,577 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69c8e4996a4c026e163483b0/lava-logs.yaml 2026-03-29 08:37:22,581 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:37:22,696 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:37:22,697 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69c8e4996a4c026e163483b0/lava_callback.json.gz 2026-03-29 08:37:22,715 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:22,730 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4996a4c026e163483b0 HTTP/1.1" 200 None 2026-03-29 08:37:22,731 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:22,745 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:37:22,747 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:22,772 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4996a4c026e163483b0 HTTP/1.1" 200 None 2026-03-29 08:37:22,773 - lava_callback - INFO - Completed processing callback for node 69c8e4996a4c026e163483b0 INFO: 172.21.0.1:44610 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:37:26,936 - lava_callback - INFO - Received callback for node 69c641b2b98d4d3c0206853f INFO: 172.21.0.1:44620 - "POST /node/69c641b2b98d4d3c0206853f HTTP/1.0" 202 Accepted 2026-03-29 08:37:26,942 - lava_callback - INFO - Starting async job submit for node 69c641b2b98d4d3c0206853f 2026-03-29 08:37:26,943 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:26,957 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c641b2b98d4d3c0206853f HTTP/1.1" 200 None 2026-03-29 08:37:26,957 - lava_callback - DEBUG - Processing node 69c641b2b98d4d3c0206853f: baseline-arm-clabbe 2026-03-29 08:37:26,958 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c641b2b98d4d3c0206853f 2026-03-29 08:37:26,958 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:26,972 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c641b2b98d4d3c0206853f HTTP/1.1" 200 None 2026-03-29 08:37:26,973 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:26,987 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63ce4b98d4d3c02067dbf HTTP/1.1" 200 None 2026-03-29 08:37:26,989 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:27,009 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c641b2b98d4d3c0206853f HTTP/1.1" 200 None 2026-03-29 08:37:27,011 - lava_callback - ERROR - Error processing callback for node 69c641b2b98d4d3c0206853f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-29 08:37:27,639 - lava_callback - INFO - Received callback for node 69c641b5b98d4d3c02068550 INFO: 172.21.0.1:44632 - "POST /node/69c641b5b98d4d3c02068550 HTTP/1.0" 202 Accepted 2026-03-29 08:37:27,644 - lava_callback - INFO - Starting async job submit for node 69c641b5b98d4d3c02068550 2026-03-29 08:37:27,645 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:27,659 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c641b5b98d4d3c02068550 HTTP/1.1" 200 None 2026-03-29 08:37:27,660 - lava_callback - DEBUG - Processing node 69c641b5b98d4d3c02068550: baseline-arm-clabbe 2026-03-29 08:37:27,660 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c641b5b98d4d3c02068550 2026-03-29 08:37:27,661 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:27,674 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c641b5b98d4d3c02068550 HTTP/1.1" 200 None 2026-03-29 08:37:27,675 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:27,681 - lava_callback - INFO - Received callback for node 69c8e4676a4c026e16348374 2026-03-29 08:37:27,687 - lava_callback - INFO - Starting async job submit for node 69c8e4676a4c026e16348374 INFO: 172.21.0.1:44644 - "POST /node/69c8e4676a4c026e16348374 HTTP/1.0" 202 Accepted 2026-03-29 08:37:27,708 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:27,715 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63ce4b98d4d3c02067dbf HTTP/1.1" 200 None 2026-03-29 08:37:27,717 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:27,732 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4676a4c026e16348374 HTTP/1.1" 200 None 2026-03-29 08:37:27,733 - lava_callback - DEBUG - Processing node 69c8e4676a4c026e16348374: baseline-x86 2026-03-29 08:37:27,796 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c641b5b98d4d3c02068550 HTTP/1.1" 200 None 2026-03-29 08:37:27,799 - lava_callback - ERROR - Error processing callback for node 69c641b5b98d4d3c02068550: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-29 08:37:28,116 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:37:28,201 - lava_callback - INFO - Received callback for node 69c8e4966a4c026e163483a3 2026-03-29 08:37:28,229 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:37:28,231 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-x86-69c8e4676a4c026e16348374/log.txt.gz INFO: 172.21.0.1:44648 - "POST /node/69c8e4966a4c026e163483a3 HTTP/1.0" 202 Accepted 2026-03-29 08:37:28,259 - lava_callback - INFO - Starting async job submit for node 69c8e4966a4c026e163483a3 2026-03-29 08:37:28,304 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:28,334 - lava_callback - INFO - Received callback for node 69c641b6b98d4d3c02068553 2026-03-29 08:37:28,344 - lava_callback - INFO - Starting async job submit for node 69c641b6b98d4d3c02068553 INFO: 172.21.0.1:44662 - "POST /node/69c641b6b98d4d3c02068553 HTTP/1.0" 202 Accepted 2026-03-29 08:37:28,360 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:28,392 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4966a4c026e163483a3 HTTP/1.1" 200 None 2026-03-29 08:37:28,409 - lava_callback - DEBUG - Processing node 69c8e4966a4c026e163483a3: baseline-arm 2026-03-29 08:37:28,490 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c641b6b98d4d3c02068553 HTTP/1.1" 200 None 2026-03-29 08:37:28,501 - lava_callback - DEBUG - Processing node 69c641b6b98d4d3c02068553: baseline-arm-clabbe 2026-03-29 08:37:28,501 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c641b6b98d4d3c02068553 2026-03-29 08:37:28,507 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:28,535 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:37:28,574 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:37:28,576 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c641b6b98d4d3c02068553 HTTP/1.1" 200 None 2026-03-29 08:37:28,577 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:28,593 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63ce4b98d4d3c02067dbf HTTP/1.1" 200 None 2026-03-29 08:37:28,594 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:28,627 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c641b6b98d4d3c02068553 HTTP/1.1" 200 None 2026-03-29 08:37:28,629 - lava_callback - ERROR - Error processing callback for node 69c641b6b98d4d3c02068553: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-29 08:37:28,849 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:37:28,851 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69c8e4966a4c026e163483a3/log.txt.gz 2026-03-29 08:37:28,862 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:37:28,888 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69c8e4676a4c026e16348374/lava-logs.yaml 2026-03-29 08:37:28,932 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:37:28,933 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:37:29,079 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:37:29,081 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69c8e4966a4c026e163483a3/lava-logs.yaml 2026-03-29 08:37:29,085 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:37:29,228 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:37:29,230 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69c8e4966a4c026e163483a3/lava_callback.json.gz 2026-03-29 08:37:29,258 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:29,268 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:37:29,270 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69c8e4676a4c026e16348374/lava_callback.json.gz 2026-03-29 08:37:29,289 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:29,291 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4966a4c026e163483a3 HTTP/1.1" 200 None 2026-03-29 08:37:29,293 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:29,310 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4676a4c026e16348374 HTTP/1.1" 200 None 2026-03-29 08:37:29,312 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:29,313 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:37:29,314 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:29,328 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e00a6a4c026e16347d61 HTTP/1.1" 200 None 2026-03-29 08:37:29,329 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:29,344 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4966a4c026e163483a3 HTTP/1.1" 200 None 2026-03-29 08:37:29,349 - lava_callback - INFO - Completed processing callback for node 69c8e4966a4c026e163483a3 2026-03-29 08:37:29,405 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4676a4c026e16348374 HTTP/1.1" 200 None 2026-03-29 08:37:29,407 - lava_callback - INFO - Completed processing callback for node 69c8e4676a4c026e16348374 2026-03-29 08:37:29,553 - lava_callback - INFO - Received callback for node 69c641b7b98d4d3c02068556 INFO: 172.21.0.1:44664 - "POST /node/69c641b7b98d4d3c02068556 HTTP/1.0" 202 Accepted 2026-03-29 08:37:29,564 - lava_callback - INFO - Starting async job submit for node 69c641b7b98d4d3c02068556 2026-03-29 08:37:29,565 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:29,586 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c641b7b98d4d3c02068556 HTTP/1.1" 200 None 2026-03-29 08:37:29,587 - lava_callback - DEBUG - Processing node 69c641b7b98d4d3c02068556: baseline-arm-clabbe 2026-03-29 08:37:29,587 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c641b7b98d4d3c02068556 2026-03-29 08:37:29,588 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:29,608 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c641b7b98d4d3c02068556 HTTP/1.1" 200 None 2026-03-29 08:37:29,610 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:29,633 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63ce4b98d4d3c02067dbf HTTP/1.1" 200 None 2026-03-29 08:37:29,636 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:29,676 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c641b7b98d4d3c02068556 HTTP/1.1" 200 None 2026-03-29 08:37:29,678 - lava_callback - ERROR - Error processing callback for node 69c641b7b98d4d3c02068556: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-29 08:37:30,242 - lava_callback - INFO - Received callback for node 69c641b9b98d4d3c02068580 INFO: 172.21.0.1:47420 - "POST /node/69c641b9b98d4d3c02068580 HTTP/1.0" 202 Accepted 2026-03-29 08:37:30,248 - lava_callback - INFO - Starting async job submit for node 69c641b9b98d4d3c02068580 2026-03-29 08:37:30,249 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:30,264 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c641b9b98d4d3c02068580 HTTP/1.1" 200 None 2026-03-29 08:37:30,265 - lava_callback - DEBUG - Processing node 69c641b9b98d4d3c02068580: baseline-arm-clabbe 2026-03-29 08:37:30,265 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c641b9b98d4d3c02068580 2026-03-29 08:37:30,265 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:30,278 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c641b9b98d4d3c02068580 HTTP/1.1" 200 None 2026-03-29 08:37:30,280 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:30,293 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63ce4b98d4d3c02067dbf HTTP/1.1" 200 None 2026-03-29 08:37:30,294 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:30,314 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c641b9b98d4d3c02068580 HTTP/1.1" 200 None 2026-03-29 08:37:30,319 - lava_callback - ERROR - Error processing callback for node 69c641b9b98d4d3c02068580: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:47424 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:37:39,138 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:37:39,155 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:56906 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41516 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:38:01,175 - lava_callback - INFO - Received callback for node 69c8e4956a4c026e1634839d INFO: 172.21.0.1:54734 - "POST /node/69c8e4956a4c026e1634839d HTTP/1.0" 202 Accepted 2026-03-29 08:38:01,181 - lava_callback - INFO - Starting async job submit for node 69c8e4956a4c026e1634839d 2026-03-29 08:38:01,202 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:01,218 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4956a4c026e1634839d HTTP/1.1" 200 None 2026-03-29 08:38:01,219 - lava_callback - DEBUG - Processing node 69c8e4956a4c026e1634839d: baseline-arm 2026-03-29 08:38:01,433 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:01,569 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:01,570 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69c8e4956a4c026e1634839d/log.txt.gz 2026-03-29 08:38:01,687 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:01,873 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:01,874 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69c8e4956a4c026e1634839d/lava-logs.yaml 2026-03-29 08:38:01,880 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:02,041 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:02,042 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69c8e4956a4c026e1634839d/lava_callback.json.gz 2026-03-29 08:38:02,064 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:02,079 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4956a4c026e1634839d HTTP/1.1" 200 None 2026-03-29 08:38:02,081 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:02,094 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:38:02,095 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:02,120 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4956a4c026e1634839d HTTP/1.1" 200 None 2026-03-29 08:38:02,121 - lava_callback - INFO - Completed processing callback for node 69c8e4956a4c026e1634839d INFO: 172.21.0.1:54742 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54746 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:38:09,157 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:09,173 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-29 08:38:10,376 - lava_callback - INFO - Received callback for node 69c8e4bb6a4c026e16348443 INFO: 172.21.0.1:33698 - "POST /node/69c8e4bb6a4c026e16348443 HTTP/1.0" 202 Accepted 2026-03-29 08:38:10,429 - lava_callback - INFO - Starting async job submit for node 69c8e4bb6a4c026e16348443 2026-03-29 08:38:10,449 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:10,465 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4bb6a4c026e16348443 HTTP/1.1" 200 None 2026-03-29 08:38:10,466 - lava_callback - DEBUG - Processing node 69c8e4bb6a4c026e16348443: baseline-arm-pengutronix 2026-03-29 08:38:10,571 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:10,704 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:10,705 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c8e4bb6a4c026e16348443/log.txt.gz 2026-03-29 08:38:10,765 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:10,922 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:10,924 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c8e4bb6a4c026e16348443/lava-logs.yaml 2026-03-29 08:38:10,927 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:11,036 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:11,038 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c8e4bb6a4c026e16348443/lava_callback.json.gz 2026-03-29 08:38:11,058 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:11,073 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4bb6a4c026e16348443 HTTP/1.1" 200 None 2026-03-29 08:38:11,075 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:11,088 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:38:11,089 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:11,113 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4bb6a4c026e16348443 HTTP/1.1" 200 None 2026-03-29 08:38:11,118 - lava_callback - INFO - Completed processing callback for node 69c8e4bb6a4c026e16348443 2026-03-29 08:38:12,961 - lava_callback - INFO - Received callback for node 69c8e4b76a4c026e16348432 INFO: 172.21.0.1:33710 - "POST /node/69c8e4b76a4c026e16348432 HTTP/1.0" 202 Accepted 2026-03-29 08:38:12,967 - lava_callback - INFO - Starting async job submit for node 69c8e4b76a4c026e16348432 2026-03-29 08:38:12,993 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:13,008 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4b76a4c026e16348432 HTTP/1.1" 200 None 2026-03-29 08:38:13,009 - lava_callback - DEBUG - Processing node 69c8e4b76a4c026e16348432: baseline-arm-pengutronix 2026-03-29 08:38:13,152 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:13,277 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:13,279 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c8e4b76a4c026e16348432/log.txt.gz 2026-03-29 08:38:13,362 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:13,545 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:13,547 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c8e4b76a4c026e16348432/lava-logs.yaml 2026-03-29 08:38:13,551 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:13,715 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:13,717 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c8e4b76a4c026e16348432/lava_callback.json.gz 2026-03-29 08:38:13,736 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:13,751 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4b76a4c026e16348432 HTTP/1.1" 200 None 2026-03-29 08:38:13,752 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:13,766 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:38:13,767 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:13,792 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4b76a4c026e16348432 HTTP/1.1" 200 None 2026-03-29 08:38:13,815 - lava_callback - INFO - Completed processing callback for node 69c8e4b76a4c026e16348432 2026-03-29 08:38:14,265 - lava_callback - INFO - Received callback for node 69c8e4b86a4c026e16348439 INFO: 172.21.0.1:33714 - "POST /node/69c8e4b86a4c026e16348439 HTTP/1.0" 202 Accepted 2026-03-29 08:38:14,271 - lava_callback - INFO - Starting async job submit for node 69c8e4b86a4c026e16348439 2026-03-29 08:38:14,294 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:14,312 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4b86a4c026e16348439 HTTP/1.1" 200 None 2026-03-29 08:38:14,313 - lava_callback - DEBUG - Processing node 69c8e4b86a4c026e16348439: baseline-arm-pengutronix 2026-03-29 08:38:14,470 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:14,590 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:14,591 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c8e4b86a4c026e16348439/log.txt.gz 2026-03-29 08:38:14,685 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:14,863 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:14,865 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c8e4b86a4c026e16348439/lava-logs.yaml 2026-03-29 08:38:14,870 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:15,024 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:15,025 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c8e4b86a4c026e16348439/lava_callback.json.gz 2026-03-29 08:38:15,046 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:15,060 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4b86a4c026e16348439 HTTP/1.1" 200 None 2026-03-29 08:38:15,062 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:15,080 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:38:15,082 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:15,107 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4b86a4c026e16348439 HTTP/1.1" 200 None 2026-03-29 08:38:15,112 - lava_callback - INFO - Completed processing callback for node 69c8e4b86a4c026e16348439 INFO: 172.21.0.1:33716 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:33724 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:38:20,455 - lava_callback - INFO - Received callback for node 69c8e4a56a4c026e163483cf INFO: 172.21.0.1:36380 - "POST /node/69c8e4a56a4c026e163483cf HTTP/1.0" 202 Accepted 2026-03-29 08:38:20,462 - lava_callback - INFO - Starting async job submit for node 69c8e4a56a4c026e163483cf 2026-03-29 08:38:20,485 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:20,500 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4a56a4c026e163483cf HTTP/1.1" 200 None 2026-03-29 08:38:20,501 - lava_callback - DEBUG - Processing node 69c8e4a56a4c026e163483cf: baseline-arm-broonie 2026-03-29 08:38:20,728 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:20,873 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:20,875 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69c8e4a56a4c026e163483cf/log.txt.gz 2026-03-29 08:38:20,998 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:21,169 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:21,171 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69c8e4a56a4c026e163483cf/lava-logs.yaml 2026-03-29 08:38:21,176 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:21,341 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:21,342 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69c8e4a56a4c026e163483cf/lava_callback.json.gz 2026-03-29 08:38:21,366 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:21,385 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4a56a4c026e163483cf HTTP/1.1" 200 None 2026-03-29 08:38:21,386 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:21,402 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:38:21,404 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:21,435 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4a56a4c026e163483cf HTTP/1.1" 200 None 2026-03-29 08:38:21,437 - lava_callback - INFO - Completed processing callback for node 69c8e4a56a4c026e163483cf 2026-03-29 08:38:21,990 - lava_callback - INFO - Received callback for node 69c8e4a76a4c026e163483d2 INFO: 172.21.0.1:36394 - "POST /node/69c8e4a76a4c026e163483d2 HTTP/1.0" 202 Accepted 2026-03-29 08:38:21,996 - lava_callback - INFO - Starting async job submit for node 69c8e4a76a4c026e163483d2 2026-03-29 08:38:22,023 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:22,036 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4a76a4c026e163483d2 HTTP/1.1" 200 None 2026-03-29 08:38:22,037 - lava_callback - DEBUG - Processing node 69c8e4a76a4c026e163483d2: baseline-arm-broonie 2026-03-29 08:38:22,249 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:22,363 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:22,365 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69c8e4a76a4c026e163483d2/log.txt.gz 2026-03-29 08:38:22,515 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:22,686 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:22,688 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69c8e4a76a4c026e163483d2/lava-logs.yaml 2026-03-29 08:38:22,694 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:22,851 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:22,852 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69c8e4a76a4c026e163483d2/lava_callback.json.gz 2026-03-29 08:38:22,876 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:22,890 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4a76a4c026e163483d2 HTTP/1.1" 200 None 2026-03-29 08:38:22,892 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:22,905 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:38:22,906 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:22,933 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4a76a4c026e163483d2 HTTP/1.1" 200 None 2026-03-29 08:38:22,935 - lava_callback - INFO - Completed processing callback for node 69c8e4a76a4c026e163483d2 2026-03-29 08:38:23,454 - lava_callback - INFO - Received callback for node 69c641f3b98d4d3c02068605 INFO: 172.21.0.1:36396 - "POST /node/69c641f3b98d4d3c02068605 HTTP/1.0" 202 Accepted 2026-03-29 08:38:23,460 - lava_callback - INFO - Starting async job submit for node 69c641f3b98d4d3c02068605 2026-03-29 08:38:23,461 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:23,479 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c641f3b98d4d3c02068605 HTTP/1.1" 200 None 2026-03-29 08:38:23,480 - lava_callback - DEBUG - Processing node 69c641f3b98d4d3c02068605: baseline-arm64-clabbe 2026-03-29 08:38:23,480 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c641f3b98d4d3c02068605 2026-03-29 08:38:23,481 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:23,499 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c641f3b98d4d3c02068605 HTTP/1.1" 200 None 2026-03-29 08:38:23,501 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:23,519 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c086b1ce0c9bc43bccb63a HTTP/1.1" 200 None 2026-03-29 08:38:23,520 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:23,540 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c641f3b98d4d3c02068605 HTTP/1.1" 200 None 2026-03-29 08:38:23,541 - lava_callback - ERROR - Error processing callback for node 69c641f3b98d4d3c02068605: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:36404 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:38:30,557 - lava_callback - INFO - Received callback for node 69c8e4ac6a4c026e163483fe INFO: 172.21.0.1:40016 - "POST /node/69c8e4ac6a4c026e163483fe HTTP/1.0" 202 Accepted 2026-03-29 08:38:30,563 - lava_callback - INFO - Starting async job submit for node 69c8e4ac6a4c026e163483fe 2026-03-29 08:38:30,585 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:30,605 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4ac6a4c026e163483fe HTTP/1.1" 200 None 2026-03-29 08:38:30,606 - lava_callback - DEBUG - Processing node 69c8e4ac6a4c026e163483fe: baseline-arm-clabbe 2026-03-29 08:38:30,830 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:30,945 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:30,947 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69c8e4ac6a4c026e163483fe/log.txt.gz 2026-03-29 08:38:31,135 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:31,335 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:31,336 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69c8e4ac6a4c026e163483fe/lava-logs.yaml 2026-03-29 08:38:31,342 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:31,479 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:31,481 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69c8e4ac6a4c026e163483fe/lava_callback.json.gz 2026-03-29 08:38:31,503 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:31,521 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4ac6a4c026e163483fe HTTP/1.1" 200 None 2026-03-29 08:38:31,522 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:31,535 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:38:31,537 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:31,561 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4ac6a4c026e163483fe HTTP/1.1" 200 None 2026-03-29 08:38:31,567 - lava_callback - INFO - Completed processing callback for node 69c8e4ac6a4c026e163483fe 2026-03-29 08:38:32,122 - lava_callback - INFO - Received callback for node 69c8e4fa6a4c026e163484d6 INFO: 172.21.0.1:40030 - "POST /node/69c8e4fa6a4c026e163484d6 HTTP/1.0" 202 Accepted 2026-03-29 08:38:32,128 - lava_callback - INFO - Starting async job submit for node 69c8e4fa6a4c026e163484d6 2026-03-29 08:38:32,131 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:32,145 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4fa6a4c026e163484d6 HTTP/1.1" 200 None 2026-03-29 08:38:32,146 - lava_callback - DEBUG - Processing node 69c8e4fa6a4c026e163484d6: baseline-x86 2026-03-29 08:38:32,154 - lava_callback - WARNING - Failed to upload log 2026-03-29 08:38:32,158 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:32,445 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:32,447 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69c8e4fa6a4c026e163484d6/lava-logs.yaml 2026-03-29 08:38:32,448 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:32,561 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:32,562 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69c8e4fa6a4c026e163484d6/lava_callback.json.gz 2026-03-29 08:38:32,565 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:32,580 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4fa6a4c026e163484d6 HTTP/1.1" 200 None 2026-03-29 08:38:32,581 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:32,594 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e00a6a4c026e16347d61 HTTP/1.1" 200 None 2026-03-29 08:38:32,596 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:32,617 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4fa6a4c026e163484d6 HTTP/1.1" 200 None 2026-03-29 08:38:32,628 - lava_callback - INFO - Completed processing callback for node 69c8e4fa6a4c026e163484d6 INFO: 172.21.0.1:40044 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:38:39,175 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:39,207 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 15 INFO: 172.21.0.1:41060 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:38:48,133 - lava_callback - INFO - Received callback for node 69c8e4fb6a4c026e163484d9 INFO: 172.21.0.1:41076 - "POST /node/69c8e4fb6a4c026e163484d9 HTTP/1.0" 202 Accepted 2026-03-29 08:38:48,141 - lava_callback - INFO - Starting async job submit for node 69c8e4fb6a4c026e163484d9 2026-03-29 08:38:48,165 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:48,187 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4fb6a4c026e163484d9 HTTP/1.1" 200 None 2026-03-29 08:38:48,188 - lava_callback - DEBUG - Processing node 69c8e4fb6a4c026e163484d9: baseline-x86 2026-03-29 08:38:48,478 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:48,671 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:48,673 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-x86-69c8e4fb6a4c026e163484d9/log.txt.gz 2026-03-29 08:38:48,797 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:49,150 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:49,153 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69c8e4fb6a4c026e163484d9/lava-logs.yaml 2026-03-29 08:38:49,159 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:49,264 - lava_callback - INFO - Received callback for node 69c8e4a86a4c026e163483d5 INFO: 172.21.0.1:41086 - "POST /node/69c8e4a86a4c026e163483d5 HTTP/1.0" 202 Accepted 2026-03-29 08:38:49,274 - lava_callback - INFO - Starting async job submit for node 69c8e4a86a4c026e163483d5 2026-03-29 08:38:49,305 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:49,328 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4a86a4c026e163483d5 HTTP/1.1" 200 None 2026-03-29 08:38:49,330 - lava_callback - DEBUG - Processing node 69c8e4a86a4c026e163483d5: baseline-arm-broonie 2026-03-29 08:38:49,358 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:49,369 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69c8e4fb6a4c026e163484d9/lava_callback.json.gz 2026-03-29 08:38:49,441 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:49,536 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4fb6a4c026e163484d9 HTTP/1.1" 200 None 2026-03-29 08:38:49,538 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:49,629 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e00a6a4c026e16347d61 HTTP/1.1" 200 None 2026-03-29 08:38:49,632 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:49,740 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4fb6a4c026e163484d9 HTTP/1.1" 200 None 2026-03-29 08:38:49,788 - lava_callback - INFO - Completed processing callback for node 69c8e4fb6a4c026e163484d9 2026-03-29 08:38:49,846 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:50,100 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:50,102 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69c8e4a86a4c026e163483d5/log.txt.gz 2026-03-29 08:38:50,321 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:50,772 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:50,775 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69c8e4a86a4c026e163483d5/lava-logs.yaml 2026-03-29 08:38:50,783 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:38:50,936 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:38:50,938 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69c8e4a86a4c026e163483d5/lava_callback.json.gz 2026-03-29 08:38:50,962 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:50,984 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4a86a4c026e163483d5 HTTP/1.1" 200 None 2026-03-29 08:38:50,986 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:50,999 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:38:51,001 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:38:51,025 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4a86a4c026e163483d5 HTTP/1.1" 200 None 2026-03-29 08:38:51,027 - lava_callback - INFO - Completed processing callback for node 69c8e4a86a4c026e163483d5 INFO: 172.21.0.1:41742 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47802 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47818 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:39:09,210 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:39:09,238 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-29 08:39:09,523 - lava_callback - INFO - Received callback for node 69c8e4a46a4c026e163483cc INFO: 172.21.0.1:47822 - "POST /node/69c8e4a46a4c026e163483cc HTTP/1.0" 202 Accepted 2026-03-29 08:39:09,534 - lava_callback - INFO - Starting async job submit for node 69c8e4a46a4c026e163483cc 2026-03-29 08:39:09,577 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:39:09,599 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4a46a4c026e163483cc HTTP/1.1" 200 None 2026-03-29 08:39:09,600 - lava_callback - DEBUG - Processing node 69c8e4a46a4c026e163483cc: baseline-arm-broonie 2026-03-29 08:39:09,841 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:39:10,341 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:39:10,343 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69c8e4a46a4c026e163483cc/log.txt.gz 2026-03-29 08:39:10,485 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:39:10,890 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:39:10,892 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69c8e4a46a4c026e163483cc/lava-logs.yaml 2026-03-29 08:39:10,900 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:39:11,124 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:39:11,126 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69c8e4a46a4c026e163483cc/lava_callback.json.gz 2026-03-29 08:39:11,150 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:39:11,188 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4a46a4c026e163483cc HTTP/1.1" 200 None 2026-03-29 08:39:11,189 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:39:11,211 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:39:11,212 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:39:11,239 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4a46a4c026e163483cc HTTP/1.1" 200 None 2026-03-29 08:39:11,242 - lava_callback - INFO - Completed processing callback for node 69c8e4a46a4c026e163483cc 2026-03-29 08:39:13,662 - lava_callback - INFO - Received callback for node 69c8e4986a4c026e163483ad INFO: 172.21.0.1:55198 - "POST /node/69c8e4986a4c026e163483ad HTTP/1.0" 202 Accepted 2026-03-29 08:39:13,671 - lava_callback - INFO - Starting async job submit for node 69c8e4986a4c026e163483ad 2026-03-29 08:39:13,708 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:39:13,738 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4986a4c026e163483ad HTTP/1.1" 200 None 2026-03-29 08:39:13,739 - lava_callback - DEBUG - Processing node 69c8e4986a4c026e163483ad: baseline-arm 2026-03-29 08:39:14,094 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:39:14,198 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:39:14,199 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69c8e4986a4c026e163483ad/log.txt.gz 2026-03-29 08:39:14,379 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:39:14,737 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:39:14,739 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69c8e4986a4c026e163483ad/lava-logs.yaml 2026-03-29 08:39:14,745 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:39:14,863 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:39:14,865 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69c8e4986a4c026e163483ad/lava_callback.json.gz 2026-03-29 08:39:14,887 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:39:14,933 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4986a4c026e163483ad HTTP/1.1" 200 None 2026-03-29 08:39:14,934 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:39:14,947 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:39:14,948 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:39:14,975 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4986a4c026e163483ad HTTP/1.1" 200 None 2026-03-29 08:39:14,977 - lava_callback - INFO - Completed processing callback for node 69c8e4986a4c026e163483ad INFO: 172.21.0.1:55212 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:55214 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39882 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41478 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:39:39,241 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:39:39,271 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:53106 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:39:56,268 - lava_callback - INFO - Received callback for node 69c8e54b6a4c026e16348534 INFO: 172.21.0.1:49630 - "POST /node/69c8e54b6a4c026e16348534 HTTP/1.0" 202 Accepted 2026-03-29 08:39:56,279 - lava_callback - INFO - Starting async job submit for node 69c8e54b6a4c026e16348534 2026-03-29 08:39:56,284 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:39:56,315 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e54b6a4c026e16348534 HTTP/1.1" 200 None 2026-03-29 08:39:56,317 - lava_callback - DEBUG - Processing node 69c8e54b6a4c026e16348534: baseline-arm 2026-03-29 08:39:56,326 - lava_callback - WARNING - Failed to upload log 2026-03-29 08:39:56,331 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:49642 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:39:56,610 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:39:56,612 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69c8e54b6a4c026e16348534/lava-logs.yaml 2026-03-29 08:39:56,613 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:39:57,016 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:39:57,018 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69c8e54b6a4c026e16348534/lava_callback.json.gz 2026-03-29 08:39:57,023 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:39:57,047 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e54b6a4c026e16348534 HTTP/1.1" 200 None 2026-03-29 08:39:57,050 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:39:57,072 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:39:57,075 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:39:57,108 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e54b6a4c026e16348534 HTTP/1.1" 200 None 2026-03-29 08:39:57,115 - lava_callback - INFO - Completed processing callback for node 69c8e54b6a4c026e16348534 INFO: 172.21.0.1:38184 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38192 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:40:09,273 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:40:09,295 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-29 08:40:13,832 - lava_callback - INFO - Received callback for node 69c8e4ad6a4c026e16348402 INFO: 172.21.0.1:44552 - "POST /node/69c8e4ad6a4c026e16348402 HTTP/1.0" 202 Accepted 2026-03-29 08:40:13,838 - lava_callback - INFO - Starting async job submit for node 69c8e4ad6a4c026e16348402 2026-03-29 08:40:13,859 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:40:13,874 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4ad6a4c026e16348402 HTTP/1.1" 200 None 2026-03-29 08:40:13,875 - lava_callback - DEBUG - Processing node 69c8e4ad6a4c026e16348402: baseline-arm-clabbe 2026-03-29 08:40:14,088 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:40:14,200 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:40:14,201 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69c8e4ad6a4c026e16348402/log.txt.gz 2026-03-29 08:40:14,322 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:40:14,501 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:40:14,503 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69c8e4ad6a4c026e16348402/lava-logs.yaml 2026-03-29 08:40:14,508 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:40:14,625 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:40:14,627 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69c8e4ad6a4c026e16348402/lava_callback.json.gz 2026-03-29 08:40:14,650 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:40:14,669 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4ad6a4c026e16348402 HTTP/1.1" 200 None 2026-03-29 08:40:14,670 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:40:14,684 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:40:14,686 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:40:14,710 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4ad6a4c026e16348402 HTTP/1.1" 200 None 2026-03-29 08:40:14,712 - lava_callback - INFO - Completed processing callback for node 69c8e4ad6a4c026e16348402 INFO: 172.21.0.1:44558 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:44572 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40748 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:40:33,086 - lava_callback - INFO - Received callback for node 69c8e4d26a4c026e16348485 INFO: 172.21.0.1:41152 - "POST /node/69c8e4d26a4c026e16348485 HTTP/1.0" 202 Accepted 2026-03-29 08:40:33,093 - lava_callback - INFO - Starting async job submit for node 69c8e4d26a4c026e16348485 2026-03-29 08:40:33,116 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:40:33,131 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4d26a4c026e16348485 HTTP/1.1" 200 None 2026-03-29 08:40:33,131 - lava_callback - DEBUG - Processing node 69c8e4d26a4c026e16348485: ltp-dio 2026-03-29 08:40:33,528 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:40:33,913 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:40:33,915 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-dio-69c8e4d26a4c026e16348485/log.txt.gz 2026-03-29 08:40:34,133 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:40:34,345 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:40:34,347 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-dio-69c8e4d26a4c026e16348485/lava-logs.yaml 2026-03-29 08:40:34,356 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:40:34,499 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:40:34,502 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-dio-69c8e4d26a4c026e16348485/lava_callback.json.gz 2026-03-29 08:40:34,527 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:40:34,541 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4d26a4c026e16348485 HTTP/1.1" 200 None 2026-03-29 08:40:34,542 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:40:34,556 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:40:34,557 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:40:34,594 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4d26a4c026e16348485 HTTP/1.1" 200 None 2026-03-29 08:40:34,596 - lava_callback - INFO - Completed processing callback for node 69c8e4d26a4c026e16348485 INFO: 172.21.0.1:41160 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:40:39,297 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:40:39,313 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-29 08:40:43,603 - lava_callback - INFO - Received callback for node 69c64281b98d4d3c0206865c INFO: 172.21.0.1:56324 - "POST /node/69c64281b98d4d3c0206865c HTTP/1.0" 202 Accepted 2026-03-29 08:40:43,609 - lava_callback - INFO - Starting async job submit for node 69c64281b98d4d3c0206865c 2026-03-29 08:40:43,610 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:40:43,625 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64281b98d4d3c0206865c HTTP/1.1" 200 None 2026-03-29 08:40:43,626 - lava_callback - DEBUG - Processing node 69c64281b98d4d3c0206865c: baseline-arm64-clabbe 2026-03-29 08:40:43,626 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64281b98d4d3c0206865c 2026-03-29 08:40:43,627 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:40:43,640 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64281b98d4d3c0206865c HTTP/1.1" 200 None 2026-03-29 08:40:43,641 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:40:43,653 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c086b1ce0c9bc43bccb63a HTTP/1.1" 200 None 2026-03-29 08:40:43,655 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:40:43,678 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64281b98d4d3c0206865c HTTP/1.1" 200 None 2026-03-29 08:40:43,679 - lava_callback - ERROR - Error processing callback for node 69c64281b98d4d3c0206865c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:56326 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42480 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51742 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51758 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47512 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47520 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49870 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48536 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:41:40,658 - lava_callback - INFO - Received callback for node 69c8e4cd6a4c026e1634847c INFO: 172.21.0.1:39536 - "POST /node/69c8e4cd6a4c026e1634847c HTTP/1.0" 202 Accepted 2026-03-29 08:41:40,668 - lava_callback - INFO - Starting async job submit for node 69c8e4cd6a4c026e1634847c 2026-03-29 08:41:40,719 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:41:40,744 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4cd6a4c026e1634847c HTTP/1.1" 200 None 2026-03-29 08:41:40,745 - lava_callback - DEBUG - Processing node 69c8e4cd6a4c026e1634847c: ltp-containers 2026-03-29 08:41:41,219 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:41:41,612 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:41:41,615 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-containers-69c8e4cd6a4c026e1634847c/log.txt.gz 2026-03-29 08:41:41,891 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:41:42,501 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:41:42,503 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-containers-69c8e4cd6a4c026e1634847c/lava-logs.yaml 2026-03-29 08:41:42,512 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:41:42,694 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:41:42,696 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-containers-69c8e4cd6a4c026e1634847c/lava_callback.json.gz 2026-03-29 08:41:42,722 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:41:42,741 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4cd6a4c026e1634847c HTTP/1.1" 200 None 2026-03-29 08:41:42,742 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:41:42,764 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:41:42,766 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:41:42,810 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4cd6a4c026e1634847c HTTP/1.1" 200 None 2026-03-29 08:41:42,812 - lava_callback - INFO - Completed processing callback for node 69c8e4cd6a4c026e1634847c INFO: 172.21.0.1:39546 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:41:48,756 - lava_callback - INFO - Received callback for node 69c8e4b16a4c026e16348413 INFO: 172.21.0.1:39550 - "POST /node/69c8e4b16a4c026e16348413 HTTP/1.0" 202 Accepted 2026-03-29 08:41:48,774 - lava_callback - INFO - Starting async job submit for node 69c8e4b16a4c026e16348413 2026-03-29 08:41:48,827 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:41:48,854 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4b16a4c026e16348413 HTTP/1.1" 200 None 2026-03-29 08:41:48,855 - lava_callback - DEBUG - Processing node 69c8e4b16a4c026e16348413: baseline-arm-clabbe 2026-03-29 08:41:49,162 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:41:49,309 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:41:49,310 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69c8e4b16a4c026e16348413/log.txt.gz 2026-03-29 08:41:49,495 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:41:49,758 - lava_callback - INFO - Received callback for node 69c8e4ce6a4c026e1634847f INFO: 172.21.0.1:38974 - "POST /node/69c8e4ce6a4c026e1634847f HTTP/1.0" 202 Accepted 2026-03-29 08:41:49,769 - lava_callback - INFO - Starting async job submit for node 69c8e4ce6a4c026e1634847f 2026-03-29 08:41:49,804 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:41:49,820 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4ce6a4c026e1634847f HTTP/1.1" 200 None 2026-03-29 08:41:49,821 - lava_callback - DEBUG - Processing node 69c8e4ce6a4c026e1634847f: ltp-controllers 2026-03-29 08:41:49,857 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:41:49,894 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69c8e4b16a4c026e16348413/lava-logs.yaml 2026-03-29 08:41:49,972 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:41:50,462 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:41:50,464 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69c8e4b16a4c026e16348413/lava_callback.json.gz 2026-03-29 08:41:50,551 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:41:50,553 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:41:50,567 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4b16a4c026e16348413 HTTP/1.1" 200 None 2026-03-29 08:41:50,569 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:41:50,588 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:41:50,590 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:41:50,616 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4b16a4c026e16348413 HTTP/1.1" 200 None 2026-03-29 08:41:50,617 - lava_callback - INFO - Completed processing callback for node 69c8e4b16a4c026e16348413 2026-03-29 08:41:50,820 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:41:50,822 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-controllers-69c8e4ce6a4c026e1634847f/log.txt.gz 2026-03-29 08:41:51,312 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:41:51,525 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:41:51,527 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-controllers-69c8e4ce6a4c026e1634847f/lava-logs.yaml 2026-03-29 08:41:51,537 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:41:51,979 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:41:51,982 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-controllers-69c8e4ce6a4c026e1634847f/lava_callback.json.gz 2026-03-29 08:41:52,036 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:41:52,066 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4ce6a4c026e1634847f HTTP/1.1" 200 None 2026-03-29 08:41:52,068 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:41:52,090 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:41:52,093 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:41:52,122 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4ce6a4c026e1634847f HTTP/1.1" 200 None 2026-03-29 08:41:52,124 - lava_callback - INFO - Completed processing callback for node 69c8e4ce6a4c026e1634847f INFO: 172.21.0.1:38982 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51016 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:42:03,735 - lava_callback - INFO - Received callback for node 69c642cfb98d4d3c0206867b INFO: 172.21.0.1:51028 - "POST /node/69c642cfb98d4d3c0206867b HTTP/1.0" 202 Accepted 2026-03-29 08:42:03,745 - lava_callback - INFO - Starting async job submit for node 69c642cfb98d4d3c0206867b 2026-03-29 08:42:03,747 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:42:03,772 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c642cfb98d4d3c0206867b HTTP/1.1" 200 None 2026-03-29 08:42:03,773 - lava_callback - DEBUG - Processing node 69c642cfb98d4d3c0206867b: baseline-arm64-clabbe 2026-03-29 08:42:03,773 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c642cfb98d4d3c0206867b 2026-03-29 08:42:03,774 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:42:03,792 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c642cfb98d4d3c0206867b HTTP/1.1" 200 None 2026-03-29 08:42:03,794 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:42:03,807 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c086b1ce0c9bc43bccb63a HTTP/1.1" 200 None 2026-03-29 08:42:03,809 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:42:03,832 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c642cfb98d4d3c0206867b HTTP/1.1" 200 None 2026-03-29 08:42:03,834 - lava_callback - ERROR - Error processing callback for node 69c642cfb98d4d3c0206867b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:51034 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:42:09,315 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:42:09,332 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-29 08:42:09,740 - lava_callback - INFO - Received callback for node 69c8e4d16a4c026e16348482 INFO: 172.21.0.1:40884 - "POST /node/69c8e4d16a4c026e16348482 HTTP/1.0" 202 Accepted 2026-03-29 08:42:09,750 - lava_callback - INFO - Starting async job submit for node 69c8e4d16a4c026e16348482 2026-03-29 08:42:09,797 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:42:09,830 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4d16a4c026e16348482 HTTP/1.1" 200 None 2026-03-29 08:42:09,831 - lava_callback - DEBUG - Processing node 69c8e4d16a4c026e16348482: ltp-cve 2026-03-29 08:42:10,325 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:42:10,653 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:42:10,655 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-cve-69c8e4d16a4c026e16348482/log.txt.gz 2026-03-29 08:42:10,965 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:42:11,418 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:42:11,420 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-cve-69c8e4d16a4c026e16348482/lava-logs.yaml 2026-03-29 08:42:11,429 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:42:11,681 - lava_callback - INFO - Received callback for node 69c8e4de6a4c026e163484a6 INFO: 172.21.0.1:40896 - "POST /node/69c8e4de6a4c026e163484a6 HTTP/1.0" 202 Accepted 2026-03-29 08:42:11,697 - lava_callback - INFO - Starting async job submit for node 69c8e4de6a4c026e163484a6 2026-03-29 08:42:11,760 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:42:11,786 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4de6a4c026e163484a6 HTTP/1.1" 200 None 2026-03-29 08:42:11,787 - lava_callback - DEBUG - Processing node 69c8e4de6a4c026e163484a6: ltp-syscalls 2026-03-29 08:42:11,896 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:42:11,904 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-cve-69c8e4d16a4c026e16348482/lava_callback.json.gz 2026-03-29 08:42:12,001 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:42:12,101 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4d16a4c026e16348482 HTTP/1.1" 200 None 2026-03-29 08:42:12,114 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:42:12,171 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:42:12,194 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:42:12,293 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4d16a4c026e16348482 HTTP/1.1" 200 None 2026-03-29 08:42:12,313 - lava_callback - INFO - Completed processing callback for node 69c8e4d16a4c026e16348482 2026-03-29 08:42:13,481 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:42:14,289 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:42:14,291 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-syscalls-69c8e4de6a4c026e163484a6/log.txt.gz 2026-03-29 08:42:15,035 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:40910 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-29 08:42:15,969 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:42:15,972 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-syscalls-69c8e4de6a4c026e163484a6/lava-logs.yaml 2026-03-29 08:42:15,995 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:40918 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:42:16,642 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:42:16,644 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-syscalls-69c8e4de6a4c026e163484a6/lava_callback.json.gz 2026-03-29 08:42:16,694 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:42:16,721 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4de6a4c026e163484a6 HTTP/1.1" 200 None 2026-03-29 08:42:16,724 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:42:16,745 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:42:16,747 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:42:16,788 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4de6a4c026e163484a6 HTTP/1.1" 200 None 2026-03-29 08:42:16,790 - lava_callback - INFO - Completed processing callback for node 69c8e4de6a4c026e163484a6 INFO: 172.21.0.1:58238 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39504 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:42:39,334 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:42:39,360 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:35416 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44918 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53774 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53780 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:43:09,521 - lava_callback - INFO - Received callback for node 69c8e4d46a4c026e16348491 INFO: 172.21.0.1:53784 - "POST /node/69c8e4d46a4c026e16348491 HTTP/1.0" 202 Accepted 2026-03-29 08:43:09,531 - lava_callback - INFO - Starting async job submit for node 69c8e4d46a4c026e16348491 2026-03-29 08:43:09,574 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:43:09,601 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4d46a4c026e16348491 HTTP/1.1" 200 None 2026-03-29 08:43:09,602 - lava_callback - DEBUG - Processing node 69c8e4d46a4c026e16348491: ltp-fs_bind 2026-03-29 08:43:10,242 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:43:10,436 - lava_callback - INFO - Received callback for node 69c64308b98d4d3c020686a0 INFO: 172.21.0.1:51572 - "POST /node/69c64308b98d4d3c020686a0 HTTP/1.0" 202 Accepted 2026-03-29 08:43:10,446 - lava_callback - INFO - Starting async job submit for node 69c64308b98d4d3c020686a0 2026-03-29 08:43:10,448 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:43:10,477 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64308b98d4d3c020686a0 HTTP/1.1" 200 None 2026-03-29 08:43:10,478 - lava_callback - DEBUG - Processing node 69c64308b98d4d3c020686a0: baseline-arm-baylibre 2026-03-29 08:43:10,478 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64308b98d4d3c020686a0 2026-03-29 08:43:10,479 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:43:10,502 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64308b98d4d3c020686a0 HTTP/1.1" 200 None 2026-03-29 08:43:10,504 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:43:10,530 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c399c1bbc022e792643e8d HTTP/1.1" 200 None 2026-03-29 08:43:10,532 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:43:10,569 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64308b98d4d3c020686a0 HTTP/1.1" 200 None 2026-03-29 08:43:10,570 - lava_callback - ERROR - Error processing callback for node 69c64308b98d4d3c020686a0: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-29 08:43:10,807 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:43:10,810 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-fs_bind-69c8e4d46a4c026e16348491/log.txt.gz 2026-03-29 08:43:11,185 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:43:11,915 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:43:11,918 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-fs_bind-69c8e4d46a4c026e16348491/lava-logs.yaml 2026-03-29 08:43:11,929 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:43:12,102 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:43:12,103 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-fs_bind-69c8e4d46a4c026e16348491/lava_callback.json.gz 2026-03-29 08:43:12,133 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:43:12,148 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4d46a4c026e16348491 HTTP/1.1" 200 None 2026-03-29 08:43:12,150 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:43:12,169 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:43:12,171 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:43:12,207 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4d46a4c026e16348491 HTTP/1.1" 200 None 2026-03-29 08:43:12,209 - lava_callback - INFO - Completed processing callback for node 69c8e4d46a4c026e16348491 INFO: 172.21.0.1:51574 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:51584 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34132 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:43:30,319 - lava_callback - INFO - Received callback for node 69c6432eb98d4d3c020686af INFO: 172.21.0.1:55426 - "POST /node/69c6432eb98d4d3c020686af HTTP/1.0" 202 Accepted 2026-03-29 08:43:30,325 - lava_callback - INFO - Starting async job submit for node 69c6432eb98d4d3c020686af 2026-03-29 08:43:30,326 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:43:30,339 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6432eb98d4d3c020686af HTTP/1.1" 200 None 2026-03-29 08:43:30,340 - lava_callback - DEBUG - Processing node 69c6432eb98d4d3c020686af: baseline-arm-baylibre 2026-03-29 08:43:30,340 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c6432eb98d4d3c020686af 2026-03-29 08:43:30,341 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:43:30,363 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6432eb98d4d3c020686af HTTP/1.1" 200 None 2026-03-29 08:43:30,364 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:43:30,378 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c399c1bbc022e792643e8d HTTP/1.1" 200 None 2026-03-29 08:43:30,379 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:43:30,400 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c6432eb98d4d3c020686af HTTP/1.1" 200 None 2026-03-29 08:43:30,401 - lava_callback - ERROR - Error processing callback for node 69c6432eb98d4d3c020686af: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-29 08:43:35,815 - lava_callback - INFO - Received callback for node 69c8e4b26a4c026e16348419 INFO: 172.21.0.1:55440 - "POST /node/69c8e4b26a4c026e16348419 HTTP/1.0" 202 Accepted 2026-03-29 08:43:35,821 - lava_callback - INFO - Starting async job submit for node 69c8e4b26a4c026e16348419 2026-03-29 08:43:35,843 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:43:35,858 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4b26a4c026e16348419 HTTP/1.1" 200 None 2026-03-29 08:43:35,859 - lava_callback - DEBUG - Processing node 69c8e4b26a4c026e16348419: baseline-arm-clabbe 2026-03-29 08:43:36,093 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:43:36,238 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:43:36,240 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69c8e4b26a4c026e16348419/log.txt.gz 2026-03-29 08:43:36,370 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:55446 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:43:36,571 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:43:36,572 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69c8e4b26a4c026e16348419/lava-logs.yaml 2026-03-29 08:43:36,578 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:43:36,704 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:43:36,705 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69c8e4b26a4c026e16348419/lava_callback.json.gz 2026-03-29 08:43:36,728 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:43:36,742 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4b26a4c026e16348419 HTTP/1.1" 200 None 2026-03-29 08:43:36,744 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:43:36,757 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:43:36,759 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:43:36,798 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4b26a4c026e16348419 HTTP/1.1" 200 None 2026-03-29 08:43:36,800 - lava_callback - INFO - Completed processing callback for node 69c8e4b26a4c026e16348419 2026-03-29 08:43:39,362 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:43:39,379 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:37798 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59046 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59900 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59906 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:44:10,363 - lava_callback - INFO - Received callback for node 69c64354b98d4d3c020686c1 INFO: 172.21.0.1:60338 - "POST /node/69c64354b98d4d3c020686c1 HTTP/1.0" 202 Accepted 2026-03-29 08:44:10,368 - lava_callback - INFO - Starting async job submit for node 69c64354b98d4d3c020686c1 2026-03-29 08:44:10,369 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:44:10,384 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64354b98d4d3c020686c1 HTTP/1.1" 200 None 2026-03-29 08:44:10,385 - lava_callback - DEBUG - Processing node 69c64354b98d4d3c020686c1: baseline-arm-baylibre 2026-03-29 08:44:10,385 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64354b98d4d3c020686c1 2026-03-29 08:44:10,386 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:44:10,401 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64354b98d4d3c020686c1 HTTP/1.1" 200 None 2026-03-29 08:44:10,402 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:44:10,598 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c399c1bbc022e792643e8d HTTP/1.1" 200 None 2026-03-29 08:44:10,599 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:44:10,624 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64354b98d4d3c020686c1 HTTP/1.1" 200 None 2026-03-29 08:44:10,633 - lava_callback - ERROR - Error processing callback for node 69c64354b98d4d3c020686c1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:60340 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:60346 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54720 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49576 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59590 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53948 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46130 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46138 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:45:09,563 - lava_callback - INFO - Received callback for node 69c8e4cb6a4c026e16348479 INFO: 172.21.0.1:46148 - "POST /node/69c8e4cb6a4c026e16348479 HTTP/1.0" 202 Accepted 2026-03-29 08:45:09,569 - lava_callback - INFO - Starting async job submit for node 69c8e4cb6a4c026e16348479 2026-03-29 08:45:09,594 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:45:09,613 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4cb6a4c026e16348479 HTTP/1.1" 200 None 2026-03-29 08:45:09,614 - lava_callback - DEBUG - Processing node 69c8e4cb6a4c026e16348479: ltp-cap_bounds 2026-03-29 08:45:09,975 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:45:10,136 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:45:10,138 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-cap_bounds-69c8e4cb6a4c026e16348479/log.txt.gz 2026-03-29 08:45:10,352 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:45:10,563 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:45:10,565 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-cap_bounds-69c8e4cb6a4c026e16348479/lava-logs.yaml 2026-03-29 08:45:10,573 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:45:10,730 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:45:10,732 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-cap_bounds-69c8e4cb6a4c026e16348479/lava_callback.json.gz 2026-03-29 08:45:10,758 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:45:10,791 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4cb6a4c026e16348479 HTTP/1.1" 200 None 2026-03-29 08:45:10,792 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:45:10,806 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:45:10,807 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:45:10,831 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4cb6a4c026e16348479 HTTP/1.1" 200 None 2026-03-29 08:45:10,837 - lava_callback - INFO - Completed processing callback for node 69c8e4cb6a4c026e16348479 INFO: 172.21.0.1:51832 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:51848 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42522 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54960 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:45:39,381 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:45:39,396 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-29 08:45:41,038 - lava_callback - INFO - Received callback for node 69c8e4d66a4c026e16348494 INFO: 172.21.0.1:50000 - "POST /node/69c8e4d66a4c026e16348494 HTTP/1.0" 202 Accepted 2026-03-29 08:45:41,045 - lava_callback - INFO - Starting async job submit for node 69c8e4d66a4c026e16348494 2026-03-29 08:45:41,070 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:45:41,116 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4d66a4c026e16348494 HTTP/1.1" 200 None 2026-03-29 08:45:41,117 - lava_callback - DEBUG - Processing node 69c8e4d66a4c026e16348494: ltp-fsx 2026-03-29 08:45:41,482 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:45:41,593 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:45:41,594 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-fsx-69c8e4d66a4c026e16348494/log.txt.gz 2026-03-29 08:45:41,801 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:45:42,024 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:45:42,026 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-fsx-69c8e4d66a4c026e16348494/lava-logs.yaml 2026-03-29 08:45:42,035 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:45:42,186 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:45:42,188 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-fsx-69c8e4d66a4c026e16348494/lava_callback.json.gz 2026-03-29 08:45:42,215 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:45:42,230 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4d66a4c026e16348494 HTTP/1.1" 200 None 2026-03-29 08:45:42,232 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:45:42,245 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:45:42,247 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:45:42,271 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4d66a4c026e16348494 HTTP/1.1" 200 None 2026-03-29 08:45:42,278 - lava_callback - INFO - Completed processing callback for node 69c8e4d66a4c026e16348494 INFO: 172.21.0.1:50004 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:45:49,632 - lava_callback - INFO - Received callback for node 69c8e4d76a4c026e16348497 INFO: 172.21.0.1:50018 - "POST /node/69c8e4d76a4c026e16348497 HTTP/1.0" 202 Accepted 2026-03-29 08:45:49,639 - lava_callback - INFO - Starting async job submit for node 69c8e4d76a4c026e16348497 2026-03-29 08:45:49,664 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:45:49,698 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4d76a4c026e16348497 HTTP/1.1" 200 None 2026-03-29 08:45:49,699 - lava_callback - DEBUG - Processing node 69c8e4d76a4c026e16348497: ltp-input 2026-03-29 08:45:50,063 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:45:50,199 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:45:50,201 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-input-69c8e4d76a4c026e16348497/log.txt.gz 2026-03-29 08:45:50,400 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:45:50,595 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:45:50,597 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-input-69c8e4d76a4c026e16348497/lava-logs.yaml 2026-03-29 08:45:50,605 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:45:50,763 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:45:50,765 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-input-69c8e4d76a4c026e16348497/lava_callback.json.gz 2026-03-29 08:45:50,790 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:45:50,804 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4d76a4c026e16348497 HTTP/1.1" 200 None 2026-03-29 08:45:50,806 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:45:50,824 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:45:50,826 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:45:50,851 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4d76a4c026e16348497 HTTP/1.1" 200 None 2026-03-29 08:45:50,852 - lava_callback - INFO - Completed processing callback for node 69c8e4d76a4c026e16348497 INFO: 172.21.0.1:58794 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48078 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48094 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:46:09,398 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:46:09,414 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:57176 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:57192 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58236 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:46:29,639 - lava_callback - INFO - Received callback for node 69c8e4d86a4c026e1634849a INFO: 172.21.0.1:58242 - "POST /node/69c8e4d86a4c026e1634849a HTTP/1.0" 202 Accepted 2026-03-29 08:46:29,646 - lava_callback - INFO - Starting async job submit for node 69c8e4d86a4c026e1634849a 2026-03-29 08:46:29,674 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:46:29,689 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4d86a4c026e1634849a HTTP/1.1" 200 None 2026-03-29 08:46:29,690 - lava_callback - DEBUG - Processing node 69c8e4d86a4c026e1634849a: ltp-ipc 2026-03-29 08:46:30,069 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:46:30,223 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:46:30,224 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-ipc-69c8e4d86a4c026e1634849a/log.txt.gz 2026-03-29 08:46:30,501 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:46:30,759 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:46:30,761 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-ipc-69c8e4d86a4c026e1634849a/lava-logs.yaml 2026-03-29 08:46:30,770 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:46:30,926 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:46:30,928 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-ipc-69c8e4d86a4c026e1634849a/lava_callback.json.gz 2026-03-29 08:46:30,954 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:46:31,030 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4d86a4c026e1634849a HTTP/1.1" 200 None 2026-03-29 08:46:31,032 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:46:31,045 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:46:31,046 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:46:31,077 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4d86a4c026e1634849a HTTP/1.1" 200 None 2026-03-29 08:46:31,079 - lava_callback - INFO - Completed processing callback for node 69c8e4d86a4c026e1634849a INFO: 172.21.0.1:50496 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:46:39,416 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:46:39,433 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-29 08:46:43,930 - lava_callback - INFO - Received callback for node 69c643edb98d4d3c0206871a INFO: 172.21.0.1:42122 - "POST /node/69c643edb98d4d3c0206871a HTTP/1.0" 202 Accepted 2026-03-29 08:46:43,936 - lava_callback - INFO - Starting async job submit for node 69c643edb98d4d3c0206871a 2026-03-29 08:46:43,937 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:46:43,950 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c643edb98d4d3c0206871a HTTP/1.1" 200 None 2026-03-29 08:46:43,951 - lava_callback - DEBUG - Processing node 69c643edb98d4d3c0206871a: baseline-arm-clabbe 2026-03-29 08:46:43,951 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c643edb98d4d3c0206871a 2026-03-29 08:46:43,952 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:46:43,966 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c643edb98d4d3c0206871a HTTP/1.1" 200 None 2026-03-29 08:46:43,967 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:46:43,986 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c399c1bbc022e792643e8d HTTP/1.1" 200 None 2026-03-29 08:46:43,988 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:46:44,017 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c643edb98d4d3c0206871a HTTP/1.1" 200 None 2026-03-29 08:46:44,019 - lava_callback - ERROR - Error processing callback for node 69c643edb98d4d3c0206871a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:42134 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34890 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36908 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36924 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:47:09,709 - lava_callback - INFO - Received callback for node 69c8e4da6a4c026e1634849d INFO: 172.21.0.1:52458 - "POST /node/69c8e4da6a4c026e1634849d HTTP/1.0" 202 Accepted 2026-03-29 08:47:09,715 - lava_callback - INFO - Starting async job submit for node 69c8e4da6a4c026e1634849d 2026-03-29 08:47:09,739 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:47:09,754 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4da6a4c026e1634849d HTTP/1.1" 200 None 2026-03-29 08:47:09,755 - lava_callback - DEBUG - Processing node 69c8e4da6a4c026e1634849d: ltp-pty 2026-03-29 08:47:10,207 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:47:10,412 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:47:10,414 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-pty-69c8e4da6a4c026e1634849d/log.txt.gz 2026-03-29 08:47:10,620 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:47:10,849 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:47:10,851 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-pty-69c8e4da6a4c026e1634849d/lava-logs.yaml 2026-03-29 08:47:10,859 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:47:11,010 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:47:11,011 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-pty-69c8e4da6a4c026e1634849d/lava_callback.json.gz 2026-03-29 08:47:11,037 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:47:11,054 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4da6a4c026e1634849d HTTP/1.1" 200 None 2026-03-29 08:47:11,056 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:47:11,069 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:47:11,070 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:47:11,094 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4da6a4c026e1634849d HTTP/1.1" 200 None 2026-03-29 08:47:11,099 - lava_callback - INFO - Completed processing callback for node 69c8e4da6a4c026e1634849d INFO: 172.21.0.1:52472 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:52488 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:47:23,902 - lava_callback - INFO - Received callback for node 69c64413b98d4d3c02068731 INFO: 172.21.0.1:52240 - "POST /node/69c64413b98d4d3c02068731 HTTP/1.0" 202 Accepted 2026-03-29 08:47:23,908 - lava_callback - INFO - Starting async job submit for node 69c64413b98d4d3c02068731 2026-03-29 08:47:23,909 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:47:23,923 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64413b98d4d3c02068731 HTTP/1.1" 200 None 2026-03-29 08:47:23,924 - lava_callback - DEBUG - Processing node 69c64413b98d4d3c02068731: baseline-arm-clabbe 2026-03-29 08:47:23,924 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64413b98d4d3c02068731 2026-03-29 08:47:23,925 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:47:23,941 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64413b98d4d3c02068731 HTTP/1.1" 200 None 2026-03-29 08:47:23,943 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:47:23,956 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c399c1bbc022e792643e8d HTTP/1.1" 200 None 2026-03-29 08:47:23,957 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:47:23,978 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64413b98d4d3c02068731 HTTP/1.1" 200 None 2026-03-29 08:47:23,980 - lava_callback - ERROR - Error processing callback for node 69c64413b98d4d3c02068731: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:52246 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48132 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:47:39,435 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:47:39,463 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:38268 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47184 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52848 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:48:03,952 - lava_callback - INFO - Received callback for node 69c6443bb98d4d3c0206873e INFO: 172.21.0.1:52850 - "POST /node/69c6443bb98d4d3c0206873e HTTP/1.0" 202 Accepted 2026-03-29 08:48:03,960 - lava_callback - INFO - Starting async job submit for node 69c6443bb98d4d3c0206873e 2026-03-29 08:48:03,962 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:48:03,985 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6443bb98d4d3c0206873e HTTP/1.1" 200 None 2026-03-29 08:48:03,986 - lava_callback - DEBUG - Processing node 69c6443bb98d4d3c0206873e: baseline-arm-clabbe 2026-03-29 08:48:03,986 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c6443bb98d4d3c0206873e 2026-03-29 08:48:03,987 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:48:04,014 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6443bb98d4d3c0206873e HTTP/1.1" 200 None 2026-03-29 08:48:04,016 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:48:04,039 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c399c1bbc022e792643e8d HTTP/1.1" 200 None 2026-03-29 08:48:04,041 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:48:04,080 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c6443bb98d4d3c0206873e HTTP/1.1" 200 None 2026-03-29 08:48:04,082 - lava_callback - ERROR - Error processing callback for node 69c6443bb98d4d3c0206873e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:52866 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58734 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:58750 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39888 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44446 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:48:44,008 - lava_callback - INFO - Received callback for node 69c64462b98d4d3c02068754 INFO: 172.21.0.1:44636 - "POST /node/69c64462b98d4d3c02068754 HTTP/1.0" 202 Accepted 2026-03-29 08:48:44,018 - lava_callback - INFO - Starting async job submit for node 69c64462b98d4d3c02068754 2026-03-29 08:48:44,019 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:48:44,042 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64462b98d4d3c02068754 HTTP/1.1" 200 None 2026-03-29 08:48:44,044 - lava_callback - DEBUG - Processing node 69c64462b98d4d3c02068754: baseline-arm-clabbe 2026-03-29 08:48:44,044 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64462b98d4d3c02068754 2026-03-29 08:48:44,045 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:48:44,068 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64462b98d4d3c02068754 HTTP/1.1" 200 None 2026-03-29 08:48:44,070 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:48:44,084 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c399c1bbc022e792643e8d HTTP/1.1" 200 None 2026-03-29 08:48:44,086 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:48:44,119 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64462b98d4d3c02068754 HTTP/1.1" 200 None 2026-03-29 08:48:44,121 - lava_callback - ERROR - Error processing callback for node 69c64462b98d4d3c02068754: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:44648 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55152 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46538 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46554 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44300 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:44308 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:49:21,121 - lava_callback - INFO - Received callback for node 69c8e2f16a4c026e16348349 INFO: 172.21.0.1:58930 - "POST /node/69c8e2f16a4c026e16348349 HTTP/1.0" 202 Accepted 2026-03-29 08:49:21,128 - lava_callback - INFO - Starting async job submit for node 69c8e2f16a4c026e16348349 2026-03-29 08:49:21,263 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:49:21,310 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e2f16a4c026e16348349 HTTP/1.1" 200 None 2026-03-29 08:49:21,311 - lava_callback - DEBUG - Processing node 69c8e2f16a4c026e16348349: ltp-smoke 2026-03-29 08:49:22,174 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:49:22,558 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:49:22,560 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-smoke-69c8e2f16a4c026e16348349/log.txt.gz 2026-03-29 08:49:23,021 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:49:24,393 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:49:24,396 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-smoke-69c8e2f16a4c026e16348349/lava-logs.yaml 2026-03-29 08:49:24,412 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:49:24,930 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:49:24,932 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-smoke-69c8e2f16a4c026e16348349/lava_callback.json.gz 2026-03-29 08:49:25,150 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:49:25,190 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e2f16a4c026e16348349 HTTP/1.1" 200 None 2026-03-29 08:49:25,192 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:49:25,213 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff06a4c026e16347d3c HTTP/1.1" 200 None 2026-03-29 08:49:25,215 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:49:25,251 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e2f16a4c026e16348349 HTTP/1.1" 200 None 2026-03-29 08:49:25,506 - lava_callback - INFO - Completed processing callback for node 69c8e2f16a4c026e16348349 INFO: 172.21.0.1:58942 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47224 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:49:39,466 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:49:39,482 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-29 08:49:41,126 - lava_callback - INFO - Received callback for node 69c8e4dd6a4c026e163484a3 INFO: 172.21.0.1:35554 - "POST /node/69c8e4dd6a4c026e163484a3 HTTP/1.0" 202 Accepted 2026-03-29 08:49:41,133 - lava_callback - INFO - Starting async job submit for node 69c8e4dd6a4c026e163484a3 2026-03-29 08:49:41,159 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:49:41,174 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4dd6a4c026e163484a3 HTTP/1.1" 200 None 2026-03-29 08:49:41,175 - lava_callback - DEBUG - Processing node 69c8e4dd6a4c026e163484a3: ltp-smoke 2026-03-29 08:49:41,718 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:49:42,167 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:49:42,169 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-smoke-69c8e4dd6a4c026e163484a3/log.txt.gz 2026-03-29 08:49:42,406 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:49:42,586 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:49:42,588 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-smoke-69c8e4dd6a4c026e163484a3/lava-logs.yaml 2026-03-29 08:49:42,597 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:49:42,741 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:49:42,743 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-smoke-69c8e4dd6a4c026e163484a3/lava_callback.json.gz 2026-03-29 08:49:42,769 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:49:42,783 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4dd6a4c026e163484a3 HTTP/1.1" 200 None 2026-03-29 08:49:42,785 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:49:42,799 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:49:42,800 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:49:42,833 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4dd6a4c026e163484a3 HTTP/1.1" 200 None 2026-03-29 08:49:42,834 - lava_callback - INFO - Completed processing callback for node 69c8e4dd6a4c026e163484a3 2026-03-29 08:49:44,101 - lava_callback - INFO - Received callback for node 69c64498b98d4d3c0206876b INFO: 172.21.0.1:35562 - "POST /node/69c64498b98d4d3c0206876b HTTP/1.0" 202 Accepted 2026-03-29 08:49:44,109 - lava_callback - INFO - Starting async job submit for node 69c64498b98d4d3c0206876b 2026-03-29 08:49:44,110 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:49:44,128 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64498b98d4d3c0206876b HTTP/1.1" 200 None 2026-03-29 08:49:44,129 - lava_callback - DEBUG - Processing node 69c64498b98d4d3c0206876b: baseline-arm64-clabbe 2026-03-29 08:49:44,129 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64498b98d4d3c0206876b 2026-03-29 08:49:44,130 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:49:44,145 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64498b98d4d3c0206876b HTTP/1.1" 200 None 2026-03-29 08:49:44,146 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:49:44,160 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c086b1ce0c9bc43bccb63a HTTP/1.1" 200 None 2026-03-29 08:49:44,161 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:49:44,182 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64498b98d4d3c0206876b HTTP/1.1" 200 None 2026-03-29 08:49:44,183 - lava_callback - ERROR - Error processing callback for node 69c64498b98d4d3c0206876b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:35578 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:49:49,724 - lava_callback - INFO - Received callback for node 69c8e4db6a4c026e163484a0 INFO: 172.21.0.1:33788 - "POST /node/69c8e4db6a4c026e163484a0 HTTP/1.0" 202 Accepted 2026-03-29 08:49:49,731 - lava_callback - INFO - Starting async job submit for node 69c8e4db6a4c026e163484a0 2026-03-29 08:49:49,771 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:49:49,807 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4db6a4c026e163484a0 HTTP/1.1" 200 None 2026-03-29 08:49:49,808 - lava_callback - DEBUG - Processing node 69c8e4db6a4c026e163484a0: ltp-sched 2026-03-29 08:49:50,251 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:49:50,890 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:49:50,892 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-sched-69c8e4db6a4c026e163484a0/log.txt.gz 2026-03-29 08:49:51,162 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:49:51,395 - lava_callback - INFO - Received callback for node 69c8e4e06a4c026e163484a9 INFO: 172.21.0.1:33800 - "POST /node/69c8e4e06a4c026e163484a9 HTTP/1.0" 202 Accepted 2026-03-29 08:49:51,402 - lava_callback - INFO - Starting async job submit for node 69c8e4e06a4c026e163484a9 2026-03-29 08:49:51,424 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:49:51,431 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:49:51,432 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-sched-69c8e4db6a4c026e163484a0/lava-logs.yaml 2026-03-29 08:49:51,441 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:49:51,451 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4e06a4c026e163484a9 HTTP/1.1" 200 None 2026-03-29 08:49:51,451 - lava_callback - DEBUG - Processing node 69c8e4e06a4c026e163484a9: ltp-syscalls-ipc 2026-03-29 08:49:52,076 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:49:52,217 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:49:52,219 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-sched-69c8e4db6a4c026e163484a0/lava_callback.json.gz 2026-03-29 08:49:52,264 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:49:52,293 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4db6a4c026e163484a0 HTTP/1.1" 200 None 2026-03-29 08:49:52,295 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:49:52,319 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:49:52,322 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:49:52,373 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4db6a4c026e163484a0 HTTP/1.1" 200 None 2026-03-29 08:49:52,375 - lava_callback - INFO - Completed processing callback for node 69c8e4db6a4c026e163484a0 2026-03-29 08:49:52,586 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:49:52,589 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-syscalls-ipc-69c8e4e06a4c026e163484a9/log.txt.gz 2026-03-29 08:49:52,920 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:49:53,110 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:49:53,112 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-syscalls-ipc-69c8e4e06a4c026e163484a9/lava-logs.yaml 2026-03-29 08:49:53,121 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:49:53,284 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:49:53,286 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-syscalls-ipc-69c8e4e06a4c026e163484a9/lava_callback.json.gz 2026-03-29 08:49:53,326 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:49:53,359 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4e06a4c026e163484a9 HTTP/1.1" 200 None 2026-03-29 08:49:53,361 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:49:53,387 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:49:53,390 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:49:53,444 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4e06a4c026e163484a9 HTTP/1.1" 200 None 2026-03-29 08:49:53,445 - lava_callback - INFO - Completed processing callback for node 69c8e4e06a4c026e163484a9 INFO: 172.21.0.1:33806 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60552 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60566 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:50:09,484 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:50:09,512 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:40864 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:40878 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35604 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59510 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35828 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:50:49,909 - lava_callback - INFO - Received callback for node 69c8e4e16a4c026e163484ac INFO: 172.21.0.1:48202 - "POST /node/69c8e4e16a4c026e163484ac HTTP/1.0" 202 Accepted 2026-03-29 08:50:49,919 - lava_callback - INFO - Starting async job submit for node 69c8e4e16a4c026e163484ac 2026-03-29 08:50:49,964 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:50:49,987 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4e16a4c026e163484ac HTTP/1.1" 200 None 2026-03-29 08:50:49,988 - lava_callback - DEBUG - Processing node 69c8e4e16a4c026e163484ac: ltp-timers 2026-03-29 08:50:50,410 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:50:50,538 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:50:50,539 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-timers-69c8e4e16a4c026e163484ac/log.txt.gz 2026-03-29 08:50:50,861 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:50:51,108 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:50:51,110 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-timers-69c8e4e16a4c026e163484ac/lava-logs.yaml 2026-03-29 08:50:51,121 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:50:51,319 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:50:51,321 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-timers-69c8e4e16a4c026e163484ac/lava_callback.json.gz 2026-03-29 08:50:51,351 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:50:51,365 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4e16a4c026e163484ac HTTP/1.1" 200 None 2026-03-29 08:50:51,367 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:50:51,380 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:50:51,382 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:50:51,408 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4e16a4c026e163484ac HTTP/1.1" 200 None 2026-03-29 08:50:51,409 - lava_callback - INFO - Completed processing callback for node 69c8e4e16a4c026e163484ac INFO: 172.21.0.1:48212 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58360 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58376 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:51:09,514 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:51:09,547 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-29 08:51:09,800 - lava_callback - INFO - Received callback for node 69c8e4e26a4c026e163484af INFO: 172.21.0.1:43426 - "POST /node/69c8e4e26a4c026e163484af HTTP/1.0" 202 Accepted 2026-03-29 08:51:09,807 - lava_callback - INFO - Starting async job submit for node 69c8e4e26a4c026e163484af 2026-03-29 08:51:09,837 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:51:09,853 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4e26a4c026e163484af HTTP/1.1" 200 None 2026-03-29 08:51:09,854 - lava_callback - DEBUG - Processing node 69c8e4e26a4c026e163484af: ltp-watchqueue 2026-03-29 08:51:10,339 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:51:10,482 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:51:10,484 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-watchqueue-69c8e4e26a4c026e163484af/log.txt.gz 2026-03-29 08:51:10,709 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:51:11,023 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:51:11,026 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-watchqueue-69c8e4e26a4c026e163484af/lava-logs.yaml 2026-03-29 08:51:11,036 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:51:11,395 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:51:11,397 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-watchqueue-69c8e4e26a4c026e163484af/lava_callback.json.gz 2026-03-29 08:51:11,427 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:51:11,444 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4e26a4c026e163484af HTTP/1.1" 200 None 2026-03-29 08:51:11,445 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:51:11,460 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:51:11,461 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:51:11,497 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4e26a4c026e163484af HTTP/1.1" 200 None 2026-03-29 08:51:11,499 - lava_callback - INFO - Completed processing callback for node 69c8e4e26a4c026e163484af INFO: 172.21.0.1:43430 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:43444 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55784 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43254 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:51:39,549 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:51:39,568 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:54158 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:51:55,585 - lava_callback - INFO - Received callback for node 69c8e8256a4c026e16348715 INFO: 172.21.0.1:45022 - "POST /node/69c8e8256a4c026e16348715 HTTP/1.0" 202 Accepted 2026-03-29 08:51:55,591 - lava_callback - INFO - Starting async job submit for node 69c8e8256a4c026e16348715 2026-03-29 08:51:55,594 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:51:55,623 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8256a4c026e16348715 HTTP/1.1" 200 None 2026-03-29 08:51:55,624 - lava_callback - DEBUG - Processing node 69c8e8256a4c026e16348715: baseline-nfs-arm64 2026-03-29 08:51:55,626 - lava_callback - WARNING - Failed to upload log 2026-03-29 08:51:55,627 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:51:55,793 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:51:55,795 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e8256a4c026e16348715/lava-logs.yaml 2026-03-29 08:51:55,796 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:51:55,904 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:51:55,905 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e8256a4c026e16348715/lava_callback.json.gz 2026-03-29 08:51:55,907 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:51:55,921 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8256a4c026e16348715 HTTP/1.1" 200 None 2026-03-29 08:51:55,923 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:51:55,936 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:51:55,938 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:51:55,958 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8256a4c026e16348715 HTTP/1.1" 200 None 2026-03-29 08:51:55,961 - lava_callback - INFO - Completed processing callback for node 69c8e8256a4c026e16348715 INFO: 172.21.0.1:45024 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:51:59,749 - lava_callback - INFO - Received callback for node 69c8e8076a4c026e163486ba INFO: 172.21.0.1:37038 - "POST /node/69c8e8076a4c026e163486ba HTTP/1.0" 202 Accepted 2026-03-29 08:51:59,755 - lava_callback - INFO - Starting async job submit for node 69c8e8076a4c026e163486ba 2026-03-29 08:51:59,774 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:51:59,788 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8076a4c026e163486ba HTTP/1.1" 200 None 2026-03-29 08:51:59,789 - lava_callback - DEBUG - Processing node 69c8e8076a4c026e163486ba: baseline-arm64 2026-03-29 08:51:59,945 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:00,117 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:00,119 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8076a4c026e163486ba/log.txt.gz 2026-03-29 08:52:00,206 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:00,432 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:00,434 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8076a4c026e163486ba/lava-logs.yaml 2026-03-29 08:52:00,439 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:00,619 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:00,621 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8076a4c026e163486ba/lava_callback.json.gz 2026-03-29 08:52:00,641 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:00,656 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8076a4c026e163486ba HTTP/1.1" 200 None 2026-03-29 08:52:00,658 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:00,671 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:52:00,673 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:00,696 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8076a4c026e163486ba HTTP/1.1" 200 None 2026-03-29 08:52:00,702 - lava_callback - INFO - Completed processing callback for node 69c8e8076a4c026e163486ba INFO: 172.21.0.1:37040 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37052 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:52:09,570 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:09,619 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-29 08:52:10,604 - lava_callback - INFO - Received callback for node 69c8e8146a4c026e163486ea INFO: 172.21.0.1:42730 - "POST /node/69c8e8146a4c026e163486ea HTTP/1.0" 202 Accepted 2026-03-29 08:52:10,619 - lava_callback - INFO - Starting async job submit for node 69c8e8146a4c026e163486ea 2026-03-29 08:52:10,652 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:10,669 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8146a4c026e163486ea HTTP/1.1" 200 None 2026-03-29 08:52:10,670 - lava_callback - DEBUG - Processing node 69c8e8146a4c026e163486ea: baseline-arm64 2026-03-29 08:52:10,856 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:10,976 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:10,977 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8146a4c026e163486ea/log.txt.gz 2026-03-29 08:52:11,068 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:11,263 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:11,265 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8146a4c026e163486ea/lava-logs.yaml 2026-03-29 08:52:11,270 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:11,447 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:11,449 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8146a4c026e163486ea/lava_callback.json.gz 2026-03-29 08:52:11,479 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:11,555 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8146a4c026e163486ea HTTP/1.1" 200 None 2026-03-29 08:52:11,557 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:11,586 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:52:11,588 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:11,627 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8146a4c026e163486ea HTTP/1.1" 200 None 2026-03-29 08:52:11,629 - lava_callback - INFO - Completed processing callback for node 69c8e8146a4c026e163486ea INFO: 172.21.0.1:42734 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:42738 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:52:22,528 - lava_callback - INFO - Received callback for node 69c8e8096a4c026e163486c0 INFO: 172.21.0.1:39216 - "POST /node/69c8e8096a4c026e163486c0 HTTP/1.0" 202 Accepted 2026-03-29 08:52:22,538 - lava_callback - INFO - Starting async job submit for node 69c8e8096a4c026e163486c0 2026-03-29 08:52:22,561 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:22,575 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8096a4c026e163486c0 HTTP/1.1" 200 None 2026-03-29 08:52:22,576 - lava_callback - DEBUG - Processing node 69c8e8096a4c026e163486c0: baseline-arm64 2026-03-29 08:52:22,859 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:23,002 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:23,004 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8096a4c026e163486c0/log.txt.gz 2026-03-29 08:52:23,215 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:23,491 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:23,491 - lava_callback - INFO - Received callback for node 69c8e8106a4c026e163486d3 INFO: 172.21.0.1:39230 - "POST /node/69c8e8106a4c026e163486d3 HTTP/1.0" 202 Accepted 2026-03-29 08:52:23,500 - lava_callback - INFO - Starting async job submit for node 69c8e8106a4c026e163486d3 2026-03-29 08:52:23,523 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:23,523 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8096a4c026e163486c0/lava-logs.yaml 2026-03-29 08:52:23,531 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:23,539 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8106a4c026e163486d3 HTTP/1.1" 200 None 2026-03-29 08:52:23,540 - lava_callback - DEBUG - Processing node 69c8e8106a4c026e163486d3: baseline-arm64 2026-03-29 08:52:23,768 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:23,826 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8096a4c026e163486c0/lava_callback.json.gz 2026-03-29 08:52:23,879 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:23,915 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:23,934 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8096a4c026e163486c0 HTTP/1.1" 200 None 2026-03-29 08:52:23,938 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:24,027 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:52:24,030 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:24,122 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8096a4c026e163486c0 HTTP/1.1" 200 None 2026-03-29 08:52:24,129 - lava_callback - INFO - Completed processing callback for node 69c8e8096a4c026e163486c0 2026-03-29 08:52:24,209 - lava_callback - INFO - Received callback for node 69c64540b98d4d3c020687c9 INFO: 172.21.0.1:39244 - "POST /node/69c64540b98d4d3c020687c9 HTTP/1.0" 202 Accepted 2026-03-29 08:52:24,218 - lava_callback - INFO - Starting async job submit for node 69c64540b98d4d3c020687c9 2026-03-29 08:52:24,221 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:24,224 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:24,226 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8106a4c026e163486d3/log.txt.gz 2026-03-29 08:52:24,351 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64540b98d4d3c020687c9 HTTP/1.1" 200 None 2026-03-29 08:52:24,358 - lava_callback - DEBUG - Processing node 69c64540b98d4d3c020687c9: baseline-arm-clabbe 2026-03-29 08:52:24,364 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64540b98d4d3c020687c9 2026-03-29 08:52:24,371 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:24,503 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64540b98d4d3c020687c9 HTTP/1.1" 200 None 2026-03-29 08:52:24,507 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:24,565 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:24,573 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c399c1bbc022e792643e8d HTTP/1.1" 200 None 2026-03-29 08:52:24,575 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:24,618 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64540b98d4d3c020687c9 HTTP/1.1" 200 None 2026-03-29 08:52:24,619 - lava_callback - ERROR - Error processing callback for node 69c64540b98d4d3c020687c9: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-29 08:52:24,955 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:24,957 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8106a4c026e163486d3/lava-logs.yaml 2026-03-29 08:52:24,967 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:25,208 - lava_callback - INFO - Received callback for node 69c8e80b6a4c026e163486c6 INFO: 172.21.0.1:39254 - "POST /node/69c8e80b6a4c026e163486c6 HTTP/1.0" 202 Accepted 2026-03-29 08:52:25,215 - lava_callback - INFO - Starting async job submit for node 69c8e80b6a4c026e163486c6 2026-03-29 08:52:25,246 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:25,262 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e80b6a4c026e163486c6 HTTP/1.1" 200 None 2026-03-29 08:52:25,262 - lava_callback - DEBUG - Processing node 69c8e80b6a4c026e163486c6: baseline-arm64 2026-03-29 08:52:25,286 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:25,323 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8106a4c026e163486d3/lava_callback.json.gz 2026-03-29 08:52:25,378 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:25,434 - lava_callback - WARNING - Failed to upload log 2026-03-29 08:52:25,459 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8106a4c026e163486d3 HTTP/1.1" 200 None 2026-03-29 08:52:25,479 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:25,509 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:25,519 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:52:25,520 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:25,553 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8106a4c026e163486d3 HTTP/1.1" 200 None 2026-03-29 08:52:25,557 - lava_callback - INFO - Completed processing callback for node 69c8e8106a4c026e163486d3 2026-03-29 08:52:25,790 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:25,792 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e80b6a4c026e163486c6/lava-logs.yaml 2026-03-29 08:52:25,796 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:25,985 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:25,987 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e80b6a4c026e163486c6/lava_callback.json.gz 2026-03-29 08:52:26,093 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:26,115 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e80b6a4c026e163486c6 HTTP/1.1" 200 None 2026-03-29 08:52:26,118 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:26,139 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:52:26,142 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:26,173 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e80b6a4c026e163486c6 HTTP/1.1" 200 None 2026-03-29 08:52:26,252 - lava_callback - INFO - Completed processing callback for node 69c8e80b6a4c026e163486c6 INFO: 172.21.0.1:39264 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:52:33,734 - lava_callback - INFO - Received callback for node 69c8e8486a4c026e1634878f INFO: 172.21.0.1:45428 - "POST /node/69c8e8486a4c026e1634878f HTTP/1.0" 202 Accepted 2026-03-29 08:52:33,740 - lava_callback - INFO - Starting async job submit for node 69c8e8486a4c026e1634878f 2026-03-29 08:52:33,744 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:33,764 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8486a4c026e1634878f HTTP/1.1" 200 None 2026-03-29 08:52:33,764 - lava_callback - DEBUG - Processing node 69c8e8486a4c026e1634878f: baseline-x86 2026-03-29 08:52:33,769 - lava_callback - WARNING - Failed to upload log 2026-03-29 08:52:33,773 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:33,891 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:33,892 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69c8e8486a4c026e1634878f/lava-logs.yaml 2026-03-29 08:52:33,894 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:34,031 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:34,033 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69c8e8486a4c026e1634878f/lava_callback.json.gz 2026-03-29 08:52:34,040 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:34,086 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8486a4c026e1634878f HTTP/1.1" 200 None 2026-03-29 08:52:34,089 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:34,114 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e00a6a4c026e16347d61 HTTP/1.1" 200 None 2026-03-29 08:52:34,116 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:34,154 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8486a4c026e1634878f HTTP/1.1" 200 None 2026-03-29 08:52:34,167 - lava_callback - INFO - Completed processing callback for node 69c8e8486a4c026e1634878f 2026-03-29 08:52:34,969 - lava_callback - INFO - Received callback for node 69c8e80f6a4c026e163486cf INFO: 172.21.0.1:45444 - "POST /node/69c8e80f6a4c026e163486cf HTTP/1.0" 202 Accepted 2026-03-29 08:52:34,981 - lava_callback - INFO - Starting async job submit for node 69c8e80f6a4c026e163486cf 2026-03-29 08:52:35,017 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:35,031 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e80f6a4c026e163486cf HTTP/1.1" 200 None 2026-03-29 08:52:35,032 - lava_callback - DEBUG - Processing node 69c8e80f6a4c026e163486cf: baseline-arm64 2026-03-29 08:52:35,291 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:35,402 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:35,404 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e80f6a4c026e163486cf/log.txt.gz 2026-03-29 08:52:35,562 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:35,743 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:35,745 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e80f6a4c026e163486cf/lava-logs.yaml 2026-03-29 08:52:35,752 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:35,889 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:35,890 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e80f6a4c026e163486cf/lava_callback.json.gz 2026-03-29 08:52:35,914 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:35,927 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e80f6a4c026e163486cf HTTP/1.1" 200 None 2026-03-29 08:52:35,929 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:35,944 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:52:35,946 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:35,970 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e80f6a4c026e163486cf HTTP/1.1" 200 None 2026-03-29 08:52:35,975 - lava_callback - INFO - Completed processing callback for node 69c8e80f6a4c026e163486cf INFO: 172.21.0.1:45458 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:52:36,728 - lava_callback - INFO - Received callback for node 69c8e8136a4c026e163486e7 INFO: 172.21.0.1:45462 - "POST /node/69c8e8136a4c026e163486e7 HTTP/1.0" 202 Accepted 2026-03-29 08:52:36,739 - lava_callback - INFO - Starting async job submit for node 69c8e8136a4c026e163486e7 2026-03-29 08:52:36,779 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:36,802 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8136a4c026e163486e7 HTTP/1.1" 200 None 2026-03-29 08:52:36,803 - lava_callback - DEBUG - Processing node 69c8e8136a4c026e163486e7: baseline-arm64 2026-03-29 08:52:37,230 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:37,348 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:37,350 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8136a4c026e163486e7/log.txt.gz 2026-03-29 08:52:37,648 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:37,842 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:37,844 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8136a4c026e163486e7/lava-logs.yaml 2026-03-29 08:52:37,852 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:37,985 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:37,987 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8136a4c026e163486e7/lava_callback.json.gz 2026-03-29 08:52:38,010 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:38,023 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8136a4c026e163486e7 HTTP/1.1" 200 None 2026-03-29 08:52:38,025 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:38,038 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:52:38,039 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:38,063 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8136a4c026e163486e7 HTTP/1.1" 200 None 2026-03-29 08:52:38,064 - lava_callback - INFO - Completed processing callback for node 69c8e8136a4c026e163486e7 2026-03-29 08:52:39,622 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:39,639 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 15 2026-03-29 08:52:41,677 - lava_callback - INFO - Received callback for node 69c8e80d6a4c026e163486c9 INFO: 172.21.0.1:45298 - "POST /node/69c8e80d6a4c026e163486c9 HTTP/1.0" 202 Accepted 2026-03-29 08:52:41,684 - lava_callback - INFO - Starting async job submit for node 69c8e80d6a4c026e163486c9 2026-03-29 08:52:41,714 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:41,730 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e80d6a4c026e163486c9 HTTP/1.1" 200 None 2026-03-29 08:52:41,731 - lava_callback - DEBUG - Processing node 69c8e80d6a4c026e163486c9: baseline-arm64 2026-03-29 08:52:41,932 - lava_callback - WARNING - Failed to upload log 2026-03-29 08:52:42,037 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:42,237 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:42,239 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e80d6a4c026e163486c9/lava-logs.yaml 2026-03-29 08:52:42,243 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:42,359 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:42,361 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e80d6a4c026e163486c9/lava_callback.json.gz 2026-03-29 08:52:42,392 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:42,406 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e80d6a4c026e163486c9 HTTP/1.1" 200 None 2026-03-29 08:52:42,408 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:42,422 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:52:42,423 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:42,443 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e80d6a4c026e163486c9 HTTP/1.1" 200 None 2026-03-29 08:52:42,506 - lava_callback - INFO - Completed processing callback for node 69c8e80d6a4c026e163486c9 INFO: 172.21.0.1:45314 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:52:51,697 - lava_callback - INFO - Received callback for node 69c8e8246a4c026e16348712 INFO: 172.21.0.1:41582 - "POST /node/69c8e8246a4c026e16348712 HTTP/1.0" 202 Accepted 2026-03-29 08:52:51,704 - lava_callback - INFO - Starting async job submit for node 69c8e8246a4c026e16348712 2026-03-29 08:52:51,728 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:51,743 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8246a4c026e16348712 HTTP/1.1" 200 None 2026-03-29 08:52:51,744 - lava_callback - DEBUG - Processing node 69c8e8246a4c026e16348712: baseline-nfs-arm64 2026-03-29 08:52:52,120 - lava_callback - INFO - Received callback for node 69c8e8226a4c026e1634870f INFO: 172.21.0.1:41594 - "POST /node/69c8e8226a4c026e1634870f HTTP/1.0" 202 Accepted 2026-03-29 08:52:52,152 - lava_callback - INFO - Starting async job submit for node 69c8e8226a4c026e1634870f 2026-03-29 08:52:52,271 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:52,346 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:52,347 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8226a4c026e1634870f HTTP/1.1" 200 None 2026-03-29 08:52:52,348 - lava_callback - DEBUG - Processing node 69c8e8226a4c026e1634870f: baseline-nfs-arm64 2026-03-29 08:52:52,610 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:52,657 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e8246a4c026e16348712/log.txt.gz 2026-03-29 08:52:52,979 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:53,091 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:53,168 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:53,170 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e8226a4c026e1634870f/log.txt.gz 2026-03-29 08:52:53,316 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:53,349 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e8246a4c026e16348712/lava-logs.yaml 2026-03-29 08:52:53,421 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:53,434 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:53,577 - lava_callback - INFO - Received callback for node 69c8e8496a4c026e1634879b INFO: 172.21.0.1:41596 - "POST /node/69c8e8496a4c026e1634879b HTTP/1.0" 202 Accepted 2026-03-29 08:52:53,585 - lava_callback - INFO - Starting async job submit for node 69c8e8496a4c026e1634879b 2026-03-29 08:52:53,591 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:53,605 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:53,605 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e8246a4c026e16348712/lava_callback.json.gz 2026-03-29 08:52:53,630 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:53,644 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8496a4c026e1634879b HTTP/1.1" 200 None 2026-03-29 08:52:53,646 - lava_callback - DEBUG - Processing node 69c8e8496a4c026e1634879b: baseline-x86 2026-03-29 08:52:53,694 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8246a4c026e16348712 HTTP/1.1" 200 None 2026-03-29 08:52:53,721 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:53,785 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:52:53,808 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:53,873 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:53,883 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e8226a4c026e1634870f/lava-logs.yaml 2026-03-29 08:52:53,886 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:53,894 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:53,910 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8246a4c026e16348712 HTTP/1.1" 200 None 2026-03-29 08:52:53,915 - lava_callback - INFO - Completed processing callback for node 69c8e8246a4c026e16348712 2026-03-29 08:52:54,093 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:54,095 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-x86-69c8e8496a4c026e1634879b/log.txt.gz 2026-03-29 08:52:54,188 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:54,218 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e8226a4c026e1634870f/lava_callback.json.gz 2026-03-29 08:52:54,260 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:54,260 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:54,276 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8226a4c026e1634870f HTTP/1.1" 200 None 2026-03-29 08:52:54,278 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:54,296 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:52:54,298 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:54,331 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8226a4c026e1634870f HTTP/1.1" 200 None 2026-03-29 08:52:54,333 - lava_callback - INFO - Completed processing callback for node 69c8e8226a4c026e1634870f 2026-03-29 08:52:54,480 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:54,483 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69c8e8496a4c026e1634879b/lava-logs.yaml 2026-03-29 08:52:54,488 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:54,651 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:54,653 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69c8e8496a4c026e1634879b/lava_callback.json.gz 2026-03-29 08:52:54,683 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:54,716 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8496a4c026e1634879b HTTP/1.1" 200 None 2026-03-29 08:52:54,718 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:54,731 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e00a6a4c026e16347d61 HTTP/1.1" 200 None 2026-03-29 08:52:54,733 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:54,757 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8496a4c026e1634879b HTTP/1.1" 200 None 2026-03-29 08:52:54,759 - lava_callback - INFO - Completed processing callback for node 69c8e8496a4c026e1634879b 2026-03-29 08:52:55,561 - lava_callback - INFO - Received callback for node 69c8e81b6a4c026e163486fa INFO: 172.21.0.1:41606 - "POST /node/69c8e81b6a4c026e163486fa HTTP/1.0" 202 Accepted 2026-03-29 08:52:55,568 - lava_callback - INFO - Starting async job submit for node 69c8e81b6a4c026e163486fa 2026-03-29 08:52:55,591 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:55,605 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e81b6a4c026e163486fa HTTP/1.1" 200 None 2026-03-29 08:52:55,606 - lava_callback - DEBUG - Processing node 69c8e81b6a4c026e163486fa: baseline-arm64 2026-03-29 08:52:55,934 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:56,049 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:56,051 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e81b6a4c026e163486fa/log.txt.gz 2026-03-29 08:52:56,227 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:56,416 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:56,418 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e81b6a4c026e163486fa/lava-logs.yaml 2026-03-29 08:52:56,425 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:41608 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:52:56,554 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:56,555 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e81b6a4c026e163486fa/lava_callback.json.gz 2026-03-29 08:52:56,577 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:56,592 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e81b6a4c026e163486fa HTTP/1.1" 200 None 2026-03-29 08:52:56,593 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:56,607 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:52:56,608 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:56,639 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e81b6a4c026e163486fa HTTP/1.1" 200 None 2026-03-29 08:52:56,646 - lava_callback - INFO - Completed processing callback for node 69c8e81b6a4c026e163486fa 2026-03-29 08:52:56,848 - lava_callback - INFO - Received callback for node 69c8e80a6a4c026e163486c3 INFO: 172.21.0.1:41622 - "POST /node/69c8e80a6a4c026e163486c3 HTTP/1.0" 202 Accepted 2026-03-29 08:52:56,855 - lava_callback - INFO - Starting async job submit for node 69c8e80a6a4c026e163486c3 2026-03-29 08:52:56,878 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:56,903 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e80a6a4c026e163486c3 HTTP/1.1" 200 None 2026-03-29 08:52:56,905 - lava_callback - DEBUG - Processing node 69c8e80a6a4c026e163486c3: baseline-arm64 2026-03-29 08:52:57,242 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:57,381 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:57,383 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e80a6a4c026e163486c3/log.txt.gz 2026-03-29 08:52:57,546 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:57,728 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:57,730 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e80a6a4c026e163486c3/lava-logs.yaml 2026-03-29 08:52:57,736 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:57,872 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:57,874 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e80a6a4c026e163486c3/lava_callback.json.gz 2026-03-29 08:52:57,897 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:57,911 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e80a6a4c026e163486c3 HTTP/1.1" 200 None 2026-03-29 08:52:57,912 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:57,932 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:52:57,934 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:57,964 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e80a6a4c026e163486c3 HTTP/1.1" 200 None 2026-03-29 08:52:57,966 - lava_callback - INFO - Completed processing callback for node 69c8e80a6a4c026e163486c3 2026-03-29 08:52:59,466 - lava_callback - INFO - Received callback for node 69c8e8156a4c026e163486ed INFO: 172.21.0.1:41630 - "POST /node/69c8e8156a4c026e163486ed HTTP/1.0" 202 Accepted 2026-03-29 08:52:59,475 - lava_callback - INFO - Starting async job submit for node 69c8e8156a4c026e163486ed 2026-03-29 08:52:59,499 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:52:59,516 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8156a4c026e163486ed HTTP/1.1" 200 None 2026-03-29 08:52:59,517 - lava_callback - DEBUG - Processing node 69c8e8156a4c026e163486ed: baseline-arm64 2026-03-29 08:52:59,750 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:52:59,861 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:52:59,863 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8156a4c026e163486ed/log.txt.gz 2026-03-29 08:53:00,001 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:00,182 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:00,185 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8156a4c026e163486ed/lava-logs.yaml 2026-03-29 08:53:00,192 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:00,355 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:00,357 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8156a4c026e163486ed/lava_callback.json.gz 2026-03-29 08:53:00,383 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:00,398 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8156a4c026e163486ed HTTP/1.1" 200 None 2026-03-29 08:53:00,400 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:00,413 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:53:00,415 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:00,445 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8156a4c026e163486ed HTTP/1.1" 200 None 2026-03-29 08:53:00,447 - lava_callback - INFO - Completed processing callback for node 69c8e8156a4c026e163486ed INFO: 172.21.0.1:39496 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39500 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:53:06,662 - lava_callback - INFO - Received callback for node 69c8e8196a4c026e163486f3 INFO: 172.21.0.1:39512 - "POST /node/69c8e8196a4c026e163486f3 HTTP/1.0" 202 Accepted 2026-03-29 08:53:06,668 - lava_callback - INFO - Starting async job submit for node 69c8e8196a4c026e163486f3 2026-03-29 08:53:06,691 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:06,705 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8196a4c026e163486f3 HTTP/1.1" 200 None 2026-03-29 08:53:06,705 - lava_callback - DEBUG - Processing node 69c8e8196a4c026e163486f3: baseline-arm64 2026-03-29 08:53:07,041 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:07,162 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:07,164 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8196a4c026e163486f3/log.txt.gz 2026-03-29 08:53:07,349 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:07,552 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:07,554 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8196a4c026e163486f3/lava-logs.yaml 2026-03-29 08:53:07,561 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:07,721 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:07,723 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8196a4c026e163486f3/lava_callback.json.gz 2026-03-29 08:53:07,746 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:07,760 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8196a4c026e163486f3 HTTP/1.1" 200 None 2026-03-29 08:53:07,762 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:07,788 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:53:07,790 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:07,815 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8196a4c026e163486f3 HTTP/1.1" 200 None 2026-03-29 08:53:07,822 - lava_callback - INFO - Completed processing callback for node 69c8e8196a4c026e163486f3 2026-03-29 08:53:09,641 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:09,659 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 15 INFO: 172.21.0.1:56388 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-29 08:53:15,512 - lava_callback - INFO - Received callback for node 69c8e8086a4c026e163486bd INFO: 172.21.0.1:56398 - "POST /node/69c8e8086a4c026e163486bd HTTP/1.0" 202 Accepted 2026-03-29 08:53:15,518 - lava_callback - INFO - Starting async job submit for node 69c8e8086a4c026e163486bd 2026-03-29 08:53:15,543 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:15,571 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8086a4c026e163486bd HTTP/1.1" 200 None 2026-03-29 08:53:15,572 - lava_callback - DEBUG - Processing node 69c8e8086a4c026e163486bd: baseline-arm64 2026-03-29 08:53:15,874 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:16,050 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:16,052 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8086a4c026e163486bd/log.txt.gz 2026-03-29 08:53:16,212 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:16,388 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:16,391 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8086a4c026e163486bd/lava-logs.yaml 2026-03-29 08:53:16,397 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:56402 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:53:16,541 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:16,542 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8086a4c026e163486bd/lava_callback.json.gz 2026-03-29 08:53:16,565 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:16,579 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8086a4c026e163486bd HTTP/1.1" 200 None 2026-03-29 08:53:16,580 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:16,606 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:53:16,607 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:16,632 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8086a4c026e163486bd HTTP/1.1" 200 None 2026-03-29 08:53:16,636 - lava_callback - INFO - Completed processing callback for node 69c8e8086a4c026e163486bd 2026-03-29 08:53:16,782 - lava_callback - INFO - Received callback for node 69c8e8726a4c026e16348817 INFO: 172.21.0.1:56418 - "POST /node/69c8e8726a4c026e16348817 HTTP/1.0" 202 Accepted 2026-03-29 08:53:16,791 - lava_callback - INFO - Starting async job submit for node 69c8e8726a4c026e16348817 2026-03-29 08:53:16,796 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:16,817 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8726a4c026e16348817 HTTP/1.1" 200 None 2026-03-29 08:53:16,818 - lava_callback - DEBUG - Processing node 69c8e8726a4c026e16348817: baseline-arm 2026-03-29 08:53:16,823 - lava_callback - WARNING - Failed to upload log 2026-03-29 08:53:16,826 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:16,963 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:16,965 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69c8e8726a4c026e16348817/lava-logs.yaml 2026-03-29 08:53:16,967 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:17,096 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:17,098 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69c8e8726a4c026e16348817/lava_callback.json.gz 2026-03-29 08:53:17,101 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:17,116 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8726a4c026e16348817 HTTP/1.1" 200 None 2026-03-29 08:53:17,118 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:17,131 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:53:17,133 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:17,154 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8726a4c026e16348817 HTTP/1.1" 200 None 2026-03-29 08:53:17,161 - lava_callback - INFO - Completed processing callback for node 69c8e8726a4c026e16348817 2026-03-29 08:53:21,397 - lava_callback - INFO - Received callback for node 69c8e8376a4c026e1634875d INFO: 172.21.0.1:44818 - "POST /node/69c8e8376a4c026e1634875d HTTP/1.0" 202 Accepted 2026-03-29 08:53:21,403 - lava_callback - INFO - Starting async job submit for node 69c8e8376a4c026e1634875d 2026-03-29 08:53:21,426 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:21,464 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8376a4c026e1634875d HTTP/1.1" 200 None 2026-03-29 08:53:21,465 - lava_callback - DEBUG - Processing node 69c8e8376a4c026e1634875d: baseline-arm64-broonie 2026-03-29 08:53:21,700 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:21,823 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:21,825 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e8376a4c026e1634875d/log.txt.gz 2026-03-29 08:53:21,846 - lava_callback - INFO - Received callback for node 69c8e8266a4c026e16348718 INFO: 172.21.0.1:44834 - "POST /node/69c8e8266a4c026e16348718 HTTP/1.0" 202 Accepted 2026-03-29 08:53:21,871 - lava_callback - INFO - Starting async job submit for node 69c8e8266a4c026e16348718 2026-03-29 08:53:21,928 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:21,989 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8266a4c026e16348718 HTTP/1.1" 200 None 2026-03-29 08:53:21,991 - lava_callback - DEBUG - Processing node 69c8e8266a4c026e16348718: baseline-nfs-arm64 2026-03-29 08:53:22,006 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:22,277 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:22,330 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e8376a4c026e1634875d/lava-logs.yaml 2026-03-29 08:53:22,347 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:22,348 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:22,487 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:22,489 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e8266a4c026e16348718/log.txt.gz 2026-03-29 08:53:22,519 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:22,559 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e8376a4c026e1634875d/lava_callback.json.gz 2026-03-29 08:53:22,603 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:22,686 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8376a4c026e1634875d HTTP/1.1" 200 None 2026-03-29 08:53:22,705 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:22,724 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:22,736 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:53:22,738 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:22,772 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8376a4c026e1634875d HTTP/1.1" 200 None 2026-03-29 08:53:22,774 - lava_callback - INFO - Completed processing callback for node 69c8e8376a4c026e1634875d 2026-03-29 08:53:22,982 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:22,984 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e8266a4c026e16348718/lava-logs.yaml 2026-03-29 08:53:22,995 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:23,151 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:23,153 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e8266a4c026e16348718/lava_callback.json.gz 2026-03-29 08:53:23,178 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:23,193 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8266a4c026e16348718 HTTP/1.1" 200 None 2026-03-29 08:53:23,194 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:23,214 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:53:23,215 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:23,239 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8266a4c026e16348718 HTTP/1.1" 200 None 2026-03-29 08:53:23,240 - lava_callback - INFO - Completed processing callback for node 69c8e8266a4c026e16348718 2026-03-29 08:53:23,887 - lava_callback - INFO - Received callback for node 69c8e8126a4c026e163486d6 INFO: 172.21.0.1:44844 - "POST /node/69c8e8126a4c026e163486d6 HTTP/1.0" 202 Accepted 2026-03-29 08:53:23,894 - lava_callback - INFO - Starting async job submit for node 69c8e8126a4c026e163486d6 2026-03-29 08:53:23,916 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:23,931 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8126a4c026e163486d6 HTTP/1.1" 200 None 2026-03-29 08:53:23,932 - lava_callback - DEBUG - Processing node 69c8e8126a4c026e163486d6: baseline-arm64 2026-03-29 08:53:24,256 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:24,398 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:24,400 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8126a4c026e163486d6/log.txt.gz 2026-03-29 08:53:24,621 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:24,816 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:24,818 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8126a4c026e163486d6/lava-logs.yaml 2026-03-29 08:53:24,825 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:24,981 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:24,983 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8126a4c026e163486d6/lava_callback.json.gz 2026-03-29 08:53:25,036 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:25,050 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8126a4c026e163486d6 HTTP/1.1" 200 None 2026-03-29 08:53:25,051 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:25,064 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:53:25,066 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:25,106 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8126a4c026e163486d6 HTTP/1.1" 200 None 2026-03-29 08:53:25,161 - lava_callback - INFO - Completed processing callback for node 69c8e8126a4c026e163486d6 2026-03-29 08:53:25,858 - lava_callback - INFO - Received callback for node 69c8e8516a4c026e163487cd INFO: 172.21.0.1:44848 - "POST /node/69c8e8516a4c026e163487cd HTTP/1.0" 202 Accepted 2026-03-29 08:53:25,868 - lava_callback - INFO - Starting async job submit for node 69c8e8516a4c026e163487cd 2026-03-29 08:53:25,898 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:25,917 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8516a4c026e163487cd HTTP/1.1" 200 None 2026-03-29 08:53:25,918 - lava_callback - DEBUG - Processing node 69c8e8516a4c026e163487cd: baseline-arm64-foundriesio 2026-03-29 08:53:26,149 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:26,304 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:26,305 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-foundriesio-69c8e8516a4c026e163487cd/log.txt.gz 2026-03-29 08:53:26,436 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:44854 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:53:26,601 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:26,603 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-foundriesio-69c8e8516a4c026e163487cd/lava-logs.yaml 2026-03-29 08:53:26,608 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:26,761 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:26,762 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-foundriesio-69c8e8516a4c026e163487cd/lava_callback.json.gz 2026-03-29 08:53:26,784 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:26,799 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8516a4c026e163487cd HTTP/1.1" 200 None 2026-03-29 08:53:26,800 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:26,813 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:53:26,815 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:26,847 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8516a4c026e163487cd HTTP/1.1" 200 None 2026-03-29 08:53:26,849 - lava_callback - INFO - Completed processing callback for node 69c8e8516a4c026e163487cd 2026-03-29 08:53:27,176 - lava_callback - INFO - Received callback for node 69c8e8526a4c026e163487d5 INFO: 172.21.0.1:44868 - "POST /node/69c8e8526a4c026e163487d5 HTTP/1.0" 202 Accepted 2026-03-29 08:53:27,188 - lava_callback - INFO - Starting async job submit for node 69c8e8526a4c026e163487d5 2026-03-29 08:53:27,232 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:27,246 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8526a4c026e163487d5 HTTP/1.1" 200 None 2026-03-29 08:53:27,247 - lava_callback - DEBUG - Processing node 69c8e8526a4c026e163487d5: baseline-arm64-foundriesio 2026-03-29 08:53:27,751 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:27,885 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:27,887 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-foundriesio-69c8e8526a4c026e163487d5/log.txt.gz 2026-03-29 08:53:28,192 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:28,414 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:28,416 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-foundriesio-69c8e8526a4c026e163487d5/lava-logs.yaml 2026-03-29 08:53:28,427 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:28,576 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:28,578 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-foundriesio-69c8e8526a4c026e163487d5/lava_callback.json.gz 2026-03-29 08:53:28,610 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:28,628 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8526a4c026e163487d5 HTTP/1.1" 200 None 2026-03-29 08:53:28,630 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:28,646 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:53:28,648 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:28,680 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8526a4c026e163487d5 HTTP/1.1" 200 None 2026-03-29 08:53:28,682 - lava_callback - INFO - Completed processing callback for node 69c8e8526a4c026e163487d5 2026-03-29 08:53:30,655 - lava_callback - INFO - Received callback for node 69c8e8206a4c026e16348709 INFO: 172.21.0.1:44804 - "POST /node/69c8e8206a4c026e16348709 HTTP/1.0" 202 Accepted 2026-03-29 08:53:30,662 - lava_callback - INFO - Starting async job submit for node 69c8e8206a4c026e16348709 2026-03-29 08:53:30,686 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:30,699 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8206a4c026e16348709 HTTP/1.1" 200 None 2026-03-29 08:53:30,700 - lava_callback - DEBUG - Processing node 69c8e8206a4c026e16348709: baseline-nfs-arm64 2026-03-29 08:53:31,053 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:31,175 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:31,176 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e8206a4c026e16348709/log.txt.gz 2026-03-29 08:53:31,366 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:31,572 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:31,574 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e8206a4c026e16348709/lava-logs.yaml 2026-03-29 08:53:31,582 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:31,718 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:31,720 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e8206a4c026e16348709/lava_callback.json.gz 2026-03-29 08:53:31,743 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:31,761 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8206a4c026e16348709 HTTP/1.1" 200 None 2026-03-29 08:53:31,762 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:31,776 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:53:31,777 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:31,801 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8206a4c026e16348709 HTTP/1.1" 200 None 2026-03-29 08:53:31,803 - lava_callback - INFO - Completed processing callback for node 69c8e8206a4c026e16348709 INFO: 172.21.0.1:44820 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:53:37,910 - lava_callback - INFO - Received callback for node 69c8e8296a4c026e1634871e INFO: 172.21.0.1:44836 - "POST /node/69c8e8296a4c026e1634871e HTTP/1.0" 202 Accepted 2026-03-29 08:53:37,916 - lava_callback - INFO - Starting async job submit for node 69c8e8296a4c026e1634871e 2026-03-29 08:53:37,939 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:37,953 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8296a4c026e1634871e HTTP/1.1" 200 None 2026-03-29 08:53:37,954 - lava_callback - DEBUG - Processing node 69c8e8296a4c026e1634871e: baseline-nfs-arm64 2026-03-29 08:53:38,303 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:38,477 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:38,479 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e8296a4c026e1634871e/log.txt.gz 2026-03-29 08:53:38,703 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:38,922 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:38,924 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e8296a4c026e1634871e/lava-logs.yaml 2026-03-29 08:53:38,933 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:39,099 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:39,100 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e8296a4c026e1634871e/lava_callback.json.gz 2026-03-29 08:53:39,124 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:39,138 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8296a4c026e1634871e HTTP/1.1" 200 None 2026-03-29 08:53:39,140 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:39,152 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:53:39,154 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:39,188 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8296a4c026e1634871e HTTP/1.1" 200 None 2026-03-29 08:53:39,189 - lava_callback - INFO - Completed processing callback for node 69c8e8296a4c026e1634871e 2026-03-29 08:53:39,662 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:39,679 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 15 2026-03-29 08:53:40,980 - lava_callback - INFO - Received callback for node 69c8e83e6a4c026e16348774 INFO: 172.21.0.1:46328 - "POST /node/69c8e83e6a4c026e16348774 HTTP/1.0" 202 Accepted 2026-03-29 08:53:40,986 - lava_callback - INFO - Starting async job submit for node 69c8e83e6a4c026e16348774 2026-03-29 08:53:41,009 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:41,024 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e83e6a4c026e16348774 HTTP/1.1" 200 None 2026-03-29 08:53:41,025 - lava_callback - DEBUG - Processing node 69c8e83e6a4c026e16348774: baseline-arm64-broonie 2026-03-29 08:53:41,262 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:41,370 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:41,372 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e83e6a4c026e16348774/log.txt.gz 2026-03-29 08:53:41,512 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:41,673 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:41,675 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e83e6a4c026e16348774/lava-logs.yaml 2026-03-29 08:53:41,681 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:41,825 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:41,827 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e83e6a4c026e16348774/lava_callback.json.gz 2026-03-29 08:53:41,850 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:41,865 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e83e6a4c026e16348774 HTTP/1.1" 200 None 2026-03-29 08:53:41,866 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:41,880 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:53:41,881 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:41,911 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e83e6a4c026e16348774 HTTP/1.1" 200 None 2026-03-29 08:53:41,912 - lava_callback - INFO - Completed processing callback for node 69c8e83e6a4c026e16348774 2026-03-29 08:53:45,140 - lava_callback - INFO - Received callback for node 69c8e82b6a4c026e16348725 INFO: 172.21.0.1:46344 - "POST /node/69c8e82b6a4c026e16348725 HTTP/1.0" 202 Accepted 2026-03-29 08:53:45,147 - lava_callback - INFO - Starting async job submit for node 69c8e82b6a4c026e16348725 2026-03-29 08:53:45,170 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:45,199 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e82b6a4c026e16348725 HTTP/1.1" 200 None 2026-03-29 08:53:45,200 - lava_callback - DEBUG - Processing node 69c8e82b6a4c026e16348725: baseline-nfs-arm64 2026-03-29 08:53:45,577 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:45,715 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:45,717 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e82b6a4c026e16348725/log.txt.gz 2026-03-29 08:53:45,956 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:46,139 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:46,141 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e82b6a4c026e16348725/lava-logs.yaml 2026-03-29 08:53:46,150 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:46,295 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:46,297 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e82b6a4c026e16348725/lava_callback.json.gz 2026-03-29 08:53:46,322 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:46,336 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e82b6a4c026e16348725 HTTP/1.1" 200 None 2026-03-29 08:53:46,337 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:46,351 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:53:46,353 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:46,377 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e82b6a4c026e16348725 HTTP/1.1" 200 None 2026-03-29 08:53:46,382 - lava_callback - INFO - Completed processing callback for node 69c8e82b6a4c026e16348725 INFO: 172.21.0.1:46360 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:53:47,191 - lava_callback - INFO - Received callback for node 69c8e81d6a4c026e16348703 INFO: 172.21.0.1:46366 - "POST /node/69c8e81d6a4c026e16348703 HTTP/1.0" 202 Accepted 2026-03-29 08:53:47,197 - lava_callback - INFO - Starting async job submit for node 69c8e81d6a4c026e16348703 2026-03-29 08:53:47,220 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:47,235 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e81d6a4c026e16348703 HTTP/1.1" 200 None 2026-03-29 08:53:47,236 - lava_callback - DEBUG - Processing node 69c8e81d6a4c026e16348703: baseline-nfs-arm64 2026-03-29 08:53:47,555 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:47,698 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:47,700 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e81d6a4c026e16348703/log.txt.gz 2026-03-29 08:53:47,884 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:48,089 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:48,091 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e81d6a4c026e16348703/lava-logs.yaml 2026-03-29 08:53:48,099 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:48,224 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:48,226 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e81d6a4c026e16348703/lava_callback.json.gz 2026-03-29 08:53:48,250 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:48,266 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e81d6a4c026e16348703 HTTP/1.1" 200 None 2026-03-29 08:53:48,268 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:48,284 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:53:48,286 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:48,310 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e81d6a4c026e16348703 HTTP/1.1" 200 None 2026-03-29 08:53:48,316 - lava_callback - INFO - Completed processing callback for node 69c8e81d6a4c026e16348703 2026-03-29 08:53:51,073 - lava_callback - INFO - Received callback for node 69c8e84d6a4c026e163487a7 INFO: 172.21.0.1:46392 - "POST /node/69c8e84d6a4c026e163487a7 HTTP/1.0" 202 Accepted 2026-03-29 08:53:51,131 - lava_callback - INFO - Starting async job submit for node 69c8e84d6a4c026e163487a7 2026-03-29 08:53:51,196 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:51,224 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e84d6a4c026e163487a7 HTTP/1.1" 200 None 2026-03-29 08:53:51,225 - lava_callback - DEBUG - Processing node 69c8e84d6a4c026e163487a7: baseline-arm64-pengutronix 2026-03-29 08:53:51,477 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:51,832 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:51,834 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69c8e84d6a4c026e163487a7/log.txt.gz 2026-03-29 08:53:51,984 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:52,223 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:52,226 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69c8e84d6a4c026e163487a7/lava-logs.yaml 2026-03-29 08:53:52,233 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:53:52,590 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:53:52,592 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69c8e84d6a4c026e163487a7/lava_callback.json.gz 2026-03-29 08:53:52,621 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:52,638 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e84d6a4c026e163487a7 HTTP/1.1" 200 None 2026-03-29 08:53:52,640 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:52,655 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:53:52,656 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:53:52,683 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e84d6a4c026e163487a7 HTTP/1.1" 200 None 2026-03-29 08:53:52,684 - lava_callback - INFO - Completed processing callback for node 69c8e84d6a4c026e163487a7 INFO: 172.21.0.1:46406 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:54:00,559 - lava_callback - INFO - Received callback for node 69c8e84f6a4c026e163487c7 INFO: 172.21.0.1:41126 - "POST /node/69c8e84f6a4c026e163487c7 HTTP/1.0" 202 Accepted 2026-03-29 08:54:00,568 - lava_callback - INFO - Starting async job submit for node 69c8e84f6a4c026e163487c7 2026-03-29 08:54:00,601 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:00,646 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e84f6a4c026e163487c7 HTTP/1.1" 200 None 2026-03-29 08:54:00,647 - lava_callback - DEBUG - Processing node 69c8e84f6a4c026e163487c7: baseline-arm64-pengutronix 2026-03-29 08:54:01,018 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:01,157 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:01,159 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69c8e84f6a4c026e163487c7/log.txt.gz 2026-03-29 08:54:01,311 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:01,725 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:01,727 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69c8e84f6a4c026e163487c7/lava-logs.yaml 2026-03-29 08:54:01,734 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:01,872 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:01,873 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69c8e84f6a4c026e163487c7/lava_callback.json.gz 2026-03-29 08:54:01,895 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:01,920 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e84f6a4c026e163487c7 HTTP/1.1" 200 None 2026-03-29 08:54:01,922 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:01,943 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:54:01,945 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:01,971 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e84f6a4c026e163487c7 HTTP/1.1" 200 None 2026-03-29 08:54:01,992 - lava_callback - INFO - Completed processing callback for node 69c8e84f6a4c026e163487c7 INFO: 172.21.0.1:41140 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:54:04,419 - lava_callback - INFO - Received callback for node 69c8e8506a4c026e163487ca INFO: 172.21.0.1:41144 - "POST /node/69c8e8506a4c026e163487ca HTTP/1.0" 202 Accepted 2026-03-29 08:54:04,430 - lava_callback - INFO - Starting async job submit for node 69c8e8506a4c026e163487ca 2026-03-29 08:54:04,469 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:04,494 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8506a4c026e163487ca HTTP/1.1" 200 None 2026-03-29 08:54:04,496 - lava_callback - DEBUG - Processing node 69c8e8506a4c026e163487ca: baseline-arm64-pengutronix 2026-03-29 08:54:04,714 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:05,080 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:05,082 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69c8e8506a4c026e163487ca/log.txt.gz 2026-03-29 08:54:05,236 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:05,710 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:05,712 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69c8e8506a4c026e163487ca/lava-logs.yaml 2026-03-29 08:54:05,718 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:05,858 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:05,860 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69c8e8506a4c026e163487ca/lava_callback.json.gz 2026-03-29 08:54:05,884 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:05,903 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8506a4c026e163487ca HTTP/1.1" 200 None 2026-03-29 08:54:05,904 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:05,924 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:54:05,926 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:05,952 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8506a4c026e163487ca HTTP/1.1" 200 None 2026-03-29 08:54:05,954 - lava_callback - INFO - Completed processing callback for node 69c8e8506a4c026e163487ca INFO: 172.21.0.1:41156 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:54:09,681 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:09,709 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 15 2026-03-29 08:54:09,811 - lava_callback - INFO - Received callback for node 69c8e8426a4c026e1634877a INFO: 172.21.0.1:37070 - "POST /node/69c8e8426a4c026e1634877a HTTP/1.0" 202 Accepted 2026-03-29 08:54:09,821 - lava_callback - INFO - Starting async job submit for node 69c8e8426a4c026e1634877a 2026-03-29 08:54:09,867 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:09,890 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8426a4c026e1634877a HTTP/1.1" 200 None 2026-03-29 08:54:09,892 - lava_callback - DEBUG - Processing node 69c8e8426a4c026e1634877a: baseline-arm64-broonie 2026-03-29 08:54:10,284 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:10,398 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:10,400 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e8426a4c026e1634877a/log.txt.gz 2026-03-29 08:54:10,676 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:10,911 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:10,913 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e8426a4c026e1634877a/lava-logs.yaml 2026-03-29 08:54:10,920 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:11,073 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:11,075 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e8426a4c026e1634877a/lava_callback.json.gz 2026-03-29 08:54:11,114 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:11,143 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8426a4c026e1634877a HTTP/1.1" 200 None 2026-03-29 08:54:11,146 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:11,189 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:54:11,191 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:11,250 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8426a4c026e1634877a HTTP/1.1" 200 None 2026-03-29 08:54:11,251 - lava_callback - INFO - Completed processing callback for node 69c8e8426a4c026e1634877a 2026-03-29 08:54:11,604 - lava_callback - INFO - Received callback for node 69c8e8486a4c026e16348795 INFO: 172.21.0.1:37086 - "POST /node/69c8e8486a4c026e16348795 HTTP/1.0" 202 Accepted 2026-03-29 08:54:11,611 - lava_callback - INFO - Starting async job submit for node 69c8e8486a4c026e16348795 2026-03-29 08:54:11,637 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:11,652 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8486a4c026e16348795 HTTP/1.1" 200 None 2026-03-29 08:54:11,653 - lava_callback - DEBUG - Processing node 69c8e8486a4c026e16348795: baseline-arm64-clabbe 2026-03-29 08:54:12,087 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:12,238 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:12,240 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-clabbe-69c8e8486a4c026e16348795/log.txt.gz 2026-03-29 08:54:12,379 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:12,856 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:12,858 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-clabbe-69c8e8486a4c026e16348795/lava-logs.yaml 2026-03-29 08:54:12,865 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:13,013 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:13,015 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-clabbe-69c8e8486a4c026e16348795/lava_callback.json.gz 2026-03-29 08:54:13,037 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:13,051 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8486a4c026e16348795 HTTP/1.1" 200 None 2026-03-29 08:54:13,053 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:13,066 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:54:13,068 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:13,099 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8486a4c026e16348795 HTTP/1.1" 200 None 2026-03-29 08:54:13,100 - lava_callback - INFO - Completed processing callback for node 69c8e8486a4c026e16348795 2026-03-29 08:54:13,489 - lava_callback - INFO - Received callback for node 69c8e4976a4c026e163483aa INFO: 172.21.0.1:37102 - "POST /node/69c8e4976a4c026e163483aa HTTP/1.0" 202 Accepted 2026-03-29 08:54:13,501 - lava_callback - INFO - Starting async job submit for node 69c8e4976a4c026e163483aa 2026-03-29 08:54:13,610 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:13,639 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4976a4c026e163483aa HTTP/1.1" 200 None 2026-03-29 08:54:13,640 - lava_callback - DEBUG - Processing node 69c8e4976a4c026e163483aa: baseline-arm 2026-03-29 08:54:14,549 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:14,655 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:14,657 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69c8e4976a4c026e163483aa/log.txt.gz INFO: 172.21.0.1:37106 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-29 08:54:15,211 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:15,439 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:15,441 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69c8e4976a4c026e163483aa/lava-logs.yaml 2026-03-29 08:54:15,459 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:15,886 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:15,889 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69c8e4976a4c026e163483aa/lava_callback.json.gz DEBUG: Setup failed for 69c8e4976a4c026e163483aa. Transitting job node result: incomplete -> fail 2026-03-29 08:54:16,011 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:16,026 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4976a4c026e163483aa HTTP/1.1" 200 None 2026-03-29 08:54:16,027 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:16,042 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:54:16,043 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:16,068 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4976a4c026e163483aa HTTP/1.1" 200 None 2026-03-29 08:54:16,186 - lava_callback - INFO - Completed processing callback for node 69c8e4976a4c026e163483aa INFO: 172.21.0.1:37110 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49964 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:54:29,834 - lava_callback - INFO - Received callback for node 69c8e83a6a4c026e16348767 INFO: 172.21.0.1:58792 - "POST /node/69c8e83a6a4c026e16348767 HTTP/1.0" 202 Accepted 2026-03-29 08:54:29,844 - lava_callback - INFO - Starting async job submit for node 69c8e83a6a4c026e16348767 2026-03-29 08:54:29,895 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:29,927 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e83a6a4c026e16348767 HTTP/1.1" 200 None 2026-03-29 08:54:29,929 - lava_callback - DEBUG - Processing node 69c8e83a6a4c026e16348767: baseline-arm64-broonie 2026-03-29 08:54:30,299 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:30,690 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:30,692 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e83a6a4c026e16348767/log.txt.gz 2026-03-29 08:54:30,948 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:31,142 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:31,144 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e83a6a4c026e16348767/lava-logs.yaml 2026-03-29 08:54:31,152 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:31,320 - lava_callback - INFO - Received callback for node 69c8e83c6a4c026e1634876e INFO: 172.21.0.1:58798 - "POST /node/69c8e83c6a4c026e1634876e HTTP/1.0" 202 Accepted 2026-03-29 08:54:31,329 - lava_callback - INFO - Starting async job submit for node 69c8e83c6a4c026e1634876e 2026-03-29 08:54:31,368 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:31,398 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e83c6a4c026e1634876e HTTP/1.1" 200 None 2026-03-29 08:54:31,399 - lava_callback - DEBUG - Processing node 69c8e83c6a4c026e1634876e: baseline-arm64-broonie 2026-03-29 08:54:31,613 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:31,620 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e83a6a4c026e16348767/lava_callback.json.gz 2026-03-29 08:54:31,711 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:31,778 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e83a6a4c026e16348767 HTTP/1.1" 200 None 2026-03-29 08:54:31,804 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:31,807 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:31,834 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:54:31,837 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:31,891 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e83a6a4c026e16348767 HTTP/1.1" 200 None 2026-03-29 08:54:31,893 - lava_callback - INFO - Completed processing callback for node 69c8e83a6a4c026e16348767 2026-03-29 08:54:32,333 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:32,335 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e83c6a4c026e1634876e/log.txt.gz 2026-03-29 08:54:32,589 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:32,796 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:32,799 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e83c6a4c026e1634876e/lava-logs.yaml 2026-03-29 08:54:32,808 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:32,922 - lava_callback - INFO - Received callback for node 69c8e8406a4c026e16348777 INFO: 172.21.0.1:58812 - "POST /node/69c8e8406a4c026e16348777 HTTP/1.0" 202 Accepted 2026-03-29 08:54:32,932 - lava_callback - INFO - Starting async job submit for node 69c8e8406a4c026e16348777 2026-03-29 08:54:32,974 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:33,005 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8406a4c026e16348777 HTTP/1.1" 200 None 2026-03-29 08:54:33,006 - lava_callback - DEBUG - Processing node 69c8e8406a4c026e16348777: baseline-arm64-broonie 2026-03-29 08:54:33,171 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:33,173 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e83c6a4c026e1634876e/lava_callback.json.gz 2026-03-29 08:54:33,254 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:33,345 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e83c6a4c026e1634876e HTTP/1.1" 200 None 2026-03-29 08:54:33,378 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:33,434 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:33,443 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:54:33,445 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:33,474 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e83c6a4c026e1634876e HTTP/1.1" 200 None 2026-03-29 08:54:33,475 - lava_callback - INFO - Completed processing callback for node 69c8e83c6a4c026e1634876e 2026-03-29 08:54:33,940 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:33,942 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e8406a4c026e16348777/log.txt.gz 2026-03-29 08:54:34,189 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:34,399 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:34,401 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e8406a4c026e16348777/lava-logs.yaml 2026-03-29 08:54:34,410 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:34,473 - lava_callback - INFO - Received callback for node 69c8e8436a4c026e1634877d INFO: 172.21.0.1:58828 - "POST /node/69c8e8436a4c026e1634877d HTTP/1.0" 202 Accepted 2026-03-29 08:54:34,483 - lava_callback - INFO - Starting async job submit for node 69c8e8436a4c026e1634877d 2026-03-29 08:54:34,540 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:34,561 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8436a4c026e1634877d HTTP/1.1" 200 None 2026-03-29 08:54:34,563 - lava_callback - DEBUG - Processing node 69c8e8436a4c026e1634877d: baseline-arm64-broonie 2026-03-29 08:54:34,832 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:34,870 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e8406a4c026e16348777/lava_callback.json.gz 2026-03-29 08:54:34,933 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:35,018 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8406a4c026e16348777 HTTP/1.1" 200 None 2026-03-29 08:54:35,020 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:35,021 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:35,040 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:54:35,042 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:35,073 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8406a4c026e16348777 HTTP/1.1" 200 None 2026-03-29 08:54:35,076 - lava_callback - INFO - Completed processing callback for node 69c8e8406a4c026e16348777 2026-03-29 08:54:35,466 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:35,468 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e8436a4c026e1634877d/log.txt.gz 2026-03-29 08:54:35,699 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:35,942 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:35,945 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e8436a4c026e1634877d/lava-logs.yaml 2026-03-29 08:54:35,953 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:36,276 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:36,278 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e8436a4c026e1634877d/lava_callback.json.gz 2026-03-29 08:54:36,317 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:36,342 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8436a4c026e1634877d HTTP/1.1" 200 None 2026-03-29 08:54:36,344 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:36,362 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:54:36,364 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:36,392 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8436a4c026e1634877d HTTP/1.1" 200 None 2026-03-29 08:54:36,398 - lava_callback - INFO - Completed processing callback for node 69c8e8436a4c026e1634877d INFO: 172.21.0.1:58830 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:54:39,711 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:39,737 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 15 2026-03-29 08:54:40,990 - lava_callback - INFO - Received callback for node 69c8e8386a4c026e16348760 2026-03-29 08:54:41,005 - lava_callback - INFO - Starting async job submit for node 69c8e8386a4c026e16348760 INFO: 172.21.0.1:39716 - "POST /node/69c8e8386a4c026e16348760 HTTP/1.0" 202 Accepted 2026-03-29 08:54:41,069 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:41,105 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8386a4c026e16348760 HTTP/1.1" 200 None 2026-03-29 08:54:41,106 - lava_callback - DEBUG - Processing node 69c8e8386a4c026e16348760: baseline-arm64-broonie 2026-03-29 08:54:41,393 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:41,502 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:41,505 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e8386a4c026e16348760/log.txt.gz 2026-03-29 08:54:41,712 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:42,026 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:42,028 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e8386a4c026e16348760/lava-logs.yaml 2026-03-29 08:54:42,034 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:42,167 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:42,168 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e8386a4c026e16348760/lava_callback.json.gz 2026-03-29 08:54:42,193 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:42,209 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8386a4c026e16348760 HTTP/1.1" 200 None 2026-03-29 08:54:42,211 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:42,226 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:54:42,228 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:42,252 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8386a4c026e16348760 HTTP/1.1" 200 None 2026-03-29 08:54:42,257 - lava_callback - INFO - Completed processing callback for node 69c8e8386a4c026e16348760 2026-03-29 08:54:44,209 - lava_callback - INFO - Received callback for node 69c8e80e6a4c026e163486cc INFO: 172.21.0.1:39718 - "POST /node/69c8e80e6a4c026e163486cc HTTP/1.0" 202 Accepted 2026-03-29 08:54:44,218 - lava_callback - INFO - Starting async job submit for node 69c8e80e6a4c026e163486cc 2026-03-29 08:54:44,240 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:44,256 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e80e6a4c026e163486cc HTTP/1.1" 200 None 2026-03-29 08:54:44,257 - lava_callback - DEBUG - Processing node 69c8e80e6a4c026e163486cc: baseline-arm64 2026-03-29 08:54:44,743 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:44,961 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:44,963 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e80e6a4c026e163486cc/log.txt.gz 2026-03-29 08:54:45,170 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:45,650 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:45,653 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e80e6a4c026e163486cc/lava-logs.yaml 2026-03-29 08:54:45,663 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:54:45,802 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:54:45,804 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e80e6a4c026e163486cc/lava_callback.json.gz 2026-03-29 08:54:45,821 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:45,836 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e80e6a4c026e163486cc HTTP/1.1" 200 None 2026-03-29 08:54:45,837 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:45,851 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:54:45,852 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:54:45,883 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e80e6a4c026e163486cc HTTP/1.1" 200 None 2026-03-29 08:54:45,893 - lava_callback - INFO - Completed processing callback for node 69c8e80e6a4c026e163486cc INFO: 172.21.0.1:39732 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50306 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43136 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43146 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:55:09,740 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:55:09,774 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:53312 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:53328 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54632 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43574 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53590 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39220 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:56:01,405 - lava_callback - INFO - Received callback for node 69c8e83d6a4c026e16348771 INFO: 172.21.0.1:47184 - "POST /node/69c8e83d6a4c026e16348771 HTTP/1.0" 202 Accepted 2026-03-29 08:56:01,411 - lava_callback - INFO - Starting async job submit for node 69c8e83d6a4c026e16348771 2026-03-29 08:56:01,437 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:56:01,452 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e83d6a4c026e16348771 HTTP/1.1" 200 None 2026-03-29 08:56:01,453 - lava_callback - DEBUG - Processing node 69c8e83d6a4c026e16348771: baseline-arm64-broonie 2026-03-29 08:56:01,734 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:56:01,882 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:56:01,884 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e83d6a4c026e16348771/log.txt.gz 2026-03-29 08:56:02,046 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:56:02,251 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:56:02,253 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e83d6a4c026e16348771/lava-logs.yaml 2026-03-29 08:56:02,260 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:56:02,424 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:56:02,426 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c8e83d6a4c026e16348771/lava_callback.json.gz 2026-03-29 08:56:02,449 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:56:02,463 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e83d6a4c026e16348771 HTTP/1.1" 200 None 2026-03-29 08:56:02,465 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:56:02,478 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:56:02,480 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:56:02,504 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e83d6a4c026e16348771 HTTP/1.1" 200 None 2026-03-29 08:56:02,506 - lava_callback - INFO - Completed processing callback for node 69c8e83d6a4c026e16348771 INFO: 172.21.0.1:47194 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47202 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:56:06,665 - lava_callback - INFO - Received callback for node 69c8e81e6a4c026e16348706 INFO: 172.21.0.1:47214 - "POST /node/69c8e81e6a4c026e16348706 HTTP/1.0" 202 Accepted 2026-03-29 08:56:06,672 - lava_callback - INFO - Starting async job submit for node 69c8e81e6a4c026e16348706 2026-03-29 08:56:06,691 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:56:06,713 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e81e6a4c026e16348706 HTTP/1.1" 200 None 2026-03-29 08:56:06,714 - lava_callback - DEBUG - Processing node 69c8e81e6a4c026e16348706: baseline-nfs-arm64 2026-03-29 08:56:07,160 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:56:07,303 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:56:07,304 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e81e6a4c026e16348706/log.txt.gz 2026-03-29 08:56:07,542 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:56:07,755 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:56:07,757 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e81e6a4c026e16348706/lava-logs.yaml 2026-03-29 08:56:07,767 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:56:07,932 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:56:07,934 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e81e6a4c026e16348706/lava_callback.json.gz 2026-03-29 08:56:07,954 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:56:07,968 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e81e6a4c026e16348706 HTTP/1.1" 200 None 2026-03-29 08:56:07,970 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:56:07,984 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:56:07,985 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:56:08,010 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e81e6a4c026e16348706 HTTP/1.1" 200 None 2026-03-29 08:56:08,012 - lava_callback - INFO - Completed processing callback for node 69c8e81e6a4c026e16348706 2026-03-29 08:56:09,777 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:56:09,794 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:40166 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:40180 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45336 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46424 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:56:37,545 - lava_callback - INFO - Received callback for node 69c8e4b96a4c026e16348440 INFO: 172.21.0.1:46434 - "POST /node/69c8e4b96a4c026e16348440 HTTP/1.0" 202 Accepted 2026-03-29 08:56:37,593 - lava_callback - INFO - Starting async job submit for node 69c8e4b96a4c026e16348440 2026-03-29 08:56:37,617 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:56:37,632 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4b96a4c026e16348440 HTTP/1.1" 200 None 2026-03-29 08:56:37,633 - lava_callback - DEBUG - Processing node 69c8e4b96a4c026e16348440: baseline-arm-pengutronix 2026-03-29 08:56:37,737 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:56:37,876 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:56:37,878 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c8e4b96a4c026e16348440/log.txt.gz 2026-03-29 08:56:37,936 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:56:38,113 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:56:38,115 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c8e4b96a4c026e16348440/lava-logs.yaml 2026-03-29 08:56:38,118 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:56:38,247 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:56:38,249 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c8e4b96a4c026e16348440/lava_callback.json.gz DEBUG: Setup failed for 69c8e4b96a4c026e16348440. Transitting job node result: incomplete -> fail 2026-03-29 08:56:38,300 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:56:38,315 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4b96a4c026e16348440 HTTP/1.1" 200 None 2026-03-29 08:56:38,316 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:56:38,330 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 08:56:38,331 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:56:38,355 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4b96a4c026e16348440 HTTP/1.1" 200 None 2026-03-29 08:56:38,399 - lava_callback - INFO - Completed processing callback for node 69c8e4b96a4c026e16348440 2026-03-29 08:56:39,795 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:56:39,811 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:50236 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49842 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46364 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46370 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46462 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:46474 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44050 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42036 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38622 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38772 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57550 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57564 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47562 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47572 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57272 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45076 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54158 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59448 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34690 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34706 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55838 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:55850 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37368 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:59:31,274 - lava_callback - INFO - Received callback for node 69c646efb98d4d3c02068879 INFO: 172.21.0.1:34166 - "POST /node/69c646efb98d4d3c02068879 HTTP/1.0" 202 Accepted 2026-03-29 08:59:31,280 - lava_callback - INFO - Starting async job submit for node 69c646efb98d4d3c02068879 2026-03-29 08:59:31,281 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:59:31,304 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c646efb98d4d3c02068879 HTTP/1.1" 200 None 2026-03-29 08:59:31,305 - lava_callback - DEBUG - Processing node 69c646efb98d4d3c02068879: baseline-arm-baylibre 2026-03-29 08:59:31,305 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c646efb98d4d3c02068879 2026-03-29 08:59:31,306 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:59:31,326 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c646efb98d4d3c02068879 HTTP/1.1" 200 None 2026-03-29 08:59:31,328 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:59:31,361 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69be544f448e074b3ffe162d HTTP/1.1" 200 None 2026-03-29 08:59:31,363 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:59:31,392 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c646efb98d4d3c02068879 HTTP/1.1" 200 None 2026-03-29 08:59:31,394 - lava_callback - ERROR - Error processing callback for node 69c646efb98d4d3c02068879: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-29 08:59:32,259 - lava_callback - INFO - Received callback for node 69c8e81c6a4c026e16348700 INFO: 172.21.0.1:34176 - "POST /node/69c8e81c6a4c026e16348700 HTTP/1.0" 202 Accepted 2026-03-29 08:59:32,266 - lava_callback - INFO - Starting async job submit for node 69c8e81c6a4c026e16348700 2026-03-29 08:59:32,302 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:59:32,317 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e81c6a4c026e16348700 HTTP/1.1" 200 None 2026-03-29 08:59:32,317 - lava_callback - DEBUG - Processing node 69c8e81c6a4c026e16348700: baseline-arm64 2026-03-29 08:59:32,707 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:59:32,897 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:59:32,898 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e81c6a4c026e16348700/log.txt.gz 2026-03-29 08:59:33,106 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:59:33,337 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:59:33,339 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e81c6a4c026e16348700/lava-logs.yaml 2026-03-29 08:59:33,348 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 08:59:33,525 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 08:59:33,527 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e81c6a4c026e16348700/lava_callback.json.gz DEBUG: Setup failed for 69c8e81c6a4c026e16348700. Transitting job node result: incomplete -> fail 2026-03-29 08:59:33,601 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:59:33,622 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e81c6a4c026e16348700 HTTP/1.1" 200 None 2026-03-29 08:59:33,624 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:59:33,638 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 08:59:33,639 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:59:33,663 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e81c6a4c026e16348700 HTTP/1.1" 200 None 2026-03-29 08:59:33,735 - lava_callback - INFO - Completed processing callback for node 69c8e81c6a4c026e16348700 INFO: 172.21.0.1:34178 - "GET / HTTP/1.0" 200 OK 2026-03-29 08:59:39,814 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 08:59:39,831 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:34404 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44478 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41738 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41744 - "GET /node/69939574cebe99dd8e0743c0%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.21.0.1:41760 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:00:11,198 - lava_callback - INFO - Received callback for node 69c64713b98d4d3c02068886 INFO: 172.21.0.1:57710 - "POST /node/69c64713b98d4d3c02068886 HTTP/1.0" 202 Accepted 2026-03-29 09:00:11,203 - lava_callback - INFO - Starting async job submit for node 69c64713b98d4d3c02068886 2026-03-29 09:00:11,204 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:00:11,218 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64713b98d4d3c02068886 HTTP/1.1" 200 None 2026-03-29 09:00:11,219 - lava_callback - DEBUG - Processing node 69c64713b98d4d3c02068886: baseline-arm-baylibre 2026-03-29 09:00:11,219 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64713b98d4d3c02068886 2026-03-29 09:00:11,220 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:00:11,260 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64713b98d4d3c02068886 HTTP/1.1" 200 None 2026-03-29 09:00:11,261 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:00:11,277 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69be544f448e074b3ffe162d HTTP/1.1" 200 None 2026-03-29 09:00:11,279 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:00:11,301 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64713b98d4d3c02068886 HTTP/1.1" 200 None 2026-03-29 09:00:11,302 - lava_callback - ERROR - Error processing callback for node 69c64713b98d4d3c02068886: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:57724 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:57738 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33798 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53580 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38466 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:00:51,243 - lava_callback - INFO - Received callback for node 69c64738b98d4d3c0206888f INFO: 172.21.0.1:35076 - "POST /node/69c64738b98d4d3c0206888f HTTP/1.0" 202 Accepted 2026-03-29 09:00:51,248 - lava_callback - INFO - Starting async job submit for node 69c64738b98d4d3c0206888f 2026-03-29 09:00:51,249 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:00:51,264 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64738b98d4d3c0206888f HTTP/1.1" 200 None 2026-03-29 09:00:51,264 - lava_callback - DEBUG - Processing node 69c64738b98d4d3c0206888f: baseline-arm-baylibre 2026-03-29 09:00:51,265 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64738b98d4d3c0206888f 2026-03-29 09:00:51,265 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:00:51,279 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64738b98d4d3c0206888f HTTP/1.1" 200 None 2026-03-29 09:00:51,280 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:00:51,293 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69be544f448e074b3ffe162d HTTP/1.1" 200 None 2026-03-29 09:00:51,295 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:00:51,315 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64738b98d4d3c0206888f HTTP/1.1" 200 None 2026-03-29 09:00:51,319 - lava_callback - ERROR - Error processing callback for node 69c64738b98d4d3c0206888f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:35088 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52148 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52162 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32776 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:32782 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44354 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46736 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46262 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48722 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52552 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52560 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39568 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:39580 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47076 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42624 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41212 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:02:51,399 - lava_callback - INFO - Received callback for node 69c647a7b98d4d3c020688ad INFO: 172.21.0.1:37056 - "POST /node/69c647a7b98d4d3c020688ad HTTP/1.0" 202 Accepted 2026-03-29 09:02:51,405 - lava_callback - INFO - Starting async job submit for node 69c647a7b98d4d3c020688ad 2026-03-29 09:02:51,405 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:02:51,425 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c647a7b98d4d3c020688ad HTTP/1.1" 200 None 2026-03-29 09:02:51,426 - lava_callback - DEBUG - Processing node 69c647a7b98d4d3c020688ad: baseline-arm-baylibre 2026-03-29 09:02:51,426 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c647a7b98d4d3c020688ad 2026-03-29 09:02:51,427 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:02:51,439 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c647a7b98d4d3c020688ad HTTP/1.1" 200 None 2026-03-29 09:02:51,441 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:02:51,454 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69be544f448e074b3ffe162d HTTP/1.1" 200 None 2026-03-29 09:02:51,456 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:02:51,476 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c647a7b98d4d3c020688ad HTTP/1.1" 200 None 2026-03-29 09:02:51,485 - lava_callback - ERROR - Error processing callback for node 69c647a7b98d4d3c020688ad: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:37064 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34728 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34742 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42242 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:42258 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47182 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48116 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57292 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:03:51,404 - lava_callback - INFO - Received callback for node 69c647efb98d4d3c020688c6 INFO: 172.21.0.1:47958 - "POST /node/69c647efb98d4d3c020688c6 HTTP/1.0" 202 Accepted 2026-03-29 09:03:51,412 - lava_callback - INFO - Starting async job submit for node 69c647efb98d4d3c020688c6 2026-03-29 09:03:51,413 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:03:51,435 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c647efb98d4d3c020688c6 HTTP/1.1" 200 None 2026-03-29 09:03:51,436 - lava_callback - DEBUG - Processing node 69c647efb98d4d3c020688c6: baseline-arm-baylibre 2026-03-29 09:03:51,436 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c647efb98d4d3c020688c6 2026-03-29 09:03:51,437 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:03:51,450 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c647efb98d4d3c020688c6 HTTP/1.1" 200 None 2026-03-29 09:03:51,451 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:03:51,465 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69be544f448e074b3ffe162d HTTP/1.1" 200 None 2026-03-29 09:03:51,467 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:03:51,495 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c647efb98d4d3c020688c6 HTTP/1.1" 200 None 2026-03-29 09:03:51,503 - lava_callback - ERROR - Error processing callback for node 69c647efb98d4d3c020688c6: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:47968 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47792 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47808 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36082 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:36098 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58492 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38698 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60978 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34088 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:04:59,436 - lava_callback - INFO - Received callback for node 69c8eb2a6a4c026e163489b0 INFO: 172.21.0.1:34102 - "POST /node/69c8eb2a6a4c026e163489b0 HTTP/1.0" 202 Accepted 2026-03-29 09:04:59,441 - lava_callback - INFO - Starting async job submit for node 69c8eb2a6a4c026e163489b0 2026-03-29 09:04:59,443 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:04:59,457 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8eb2a6a4c026e163489b0 HTTP/1.1" 200 None 2026-03-29 09:04:59,458 - lava_callback - DEBUG - Processing node 69c8eb2a6a4c026e163489b0: baseline-nfs-arm64 2026-03-29 09:04:59,459 - lava_callback - WARNING - Failed to upload log 2026-03-29 09:04:59,460 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:04:59,630 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:04:59,632 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8eb2a6a4c026e163489b0/lava-logs.yaml 2026-03-29 09:04:59,633 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:04:59,769 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:04:59,771 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8eb2a6a4c026e163489b0/lava_callback.json.gz 2026-03-29 09:04:59,772 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:04:59,787 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8eb2a6a4c026e163489b0 HTTP/1.1" 200 None 2026-03-29 09:04:59,789 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:04:59,803 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 09:04:59,804 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:04:59,837 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8eb2a6a4c026e163489b0 HTTP/1.1" 200 None 2026-03-29 09:04:59,859 - lava_callback - INFO - Completed processing callback for node 69c8eb2a6a4c026e163489b0 INFO: 172.21.0.1:39726 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39728 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:05:09,834 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:05:09,850 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-29 09:05:11,537 - lava_callback - INFO - Received callback for node 69c64838b98d4d3c020688dc INFO: 172.21.0.1:47524 - "POST /node/69c64838b98d4d3c020688dc HTTP/1.0" 202 Accepted 2026-03-29 09:05:11,542 - lava_callback - INFO - Starting async job submit for node 69c64838b98d4d3c020688dc 2026-03-29 09:05:11,543 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:05:11,568 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64838b98d4d3c020688dc HTTP/1.1" 200 None 2026-03-29 09:05:11,569 - lava_callback - DEBUG - Processing node 69c64838b98d4d3c020688dc: baseline-arm-baylibre 2026-03-29 09:05:11,569 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64838b98d4d3c020688dc 2026-03-29 09:05:11,570 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:05:11,583 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64838b98d4d3c020688dc HTTP/1.1" 200 None 2026-03-29 09:05:11,584 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:05:11,597 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69be544f448e074b3ffe162d HTTP/1.1" 200 None 2026-03-29 09:05:11,599 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:05:11,619 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64838b98d4d3c020688dc HTTP/1.1" 200 None 2026-03-29 09:05:11,625 - lava_callback - ERROR - Error processing callback for node 69c64838b98d4d3c020688dc: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:47536 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47540 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60362 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50420 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48074 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36462 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56026 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56038 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44578 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:44584 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43942 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:06:31,586 - lava_callback - INFO - Received callback for node 69c64887b98d4d3c020688f7 INFO: 172.21.0.1:44688 - "POST /node/69c64887b98d4d3c020688f7 HTTP/1.0" 202 Accepted 2026-03-29 09:06:31,592 - lava_callback - INFO - Starting async job submit for node 69c64887b98d4d3c020688f7 2026-03-29 09:06:31,593 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:06:31,614 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64887b98d4d3c020688f7 HTTP/1.1" 200 None 2026-03-29 09:06:31,615 - lava_callback - DEBUG - Processing node 69c64887b98d4d3c020688f7: baseline-arm64-baylibre 2026-03-29 09:06:31,615 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64887b98d4d3c020688f7 2026-03-29 09:06:31,616 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:06:31,629 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64887b98d4d3c020688f7 HTTP/1.1" 200 None 2026-03-29 09:06:31,630 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:06:31,644 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c399cdbbc022e792643ea0 HTTP/1.1" 200 None 2026-03-29 09:06:31,645 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:06:31,666 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64887b98d4d3c020688f7 HTTP/1.1" 200 None 2026-03-29 09:06:31,673 - lava_callback - ERROR - Error processing callback for node 69c64887b98d4d3c020688f7: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:44696 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54488 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44002 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59626 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59634 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42742 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:42752 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53336 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46590 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52662 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45286 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53828 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53842 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:08:11,642 - lava_callback - INFO - Received callback for node 69c648f5b98d4d3c0206890a INFO: 172.21.0.1:42634 - "POST /node/69c648f5b98d4d3c0206890a HTTP/1.0" 202 Accepted 2026-03-29 09:08:11,649 - lava_callback - INFO - Starting async job submit for node 69c648f5b98d4d3c0206890a 2026-03-29 09:08:11,650 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:08:11,670 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c648f5b98d4d3c0206890a HTTP/1.1" 200 None 2026-03-29 09:08:11,671 - lava_callback - DEBUG - Processing node 69c648f5b98d4d3c0206890a: baseline-arm64-baylibre 2026-03-29 09:08:11,671 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c648f5b98d4d3c0206890a 2026-03-29 09:08:11,672 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:08:11,691 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c648f5b98d4d3c0206890a HTTP/1.1" 200 None 2026-03-29 09:08:11,693 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:08:11,706 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c399cdbbc022e792643ea0 HTTP/1.1" 200 None 2026-03-29 09:08:11,708 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:08:11,728 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c648f5b98d4d3c0206890a HTTP/1.1" 200 None 2026-03-29 09:08:11,729 - lava_callback - ERROR - Error processing callback for node 69c648f5b98d4d3c0206890a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:42642 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:42658 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48234 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41460 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47050 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:08:47,767 - lava_callback - INFO - Received callback for node 69c8ebdc6a4c026e163489d3 INFO: 172.21.0.1:47062 - "POST /node/69c8ebdc6a4c026e163489d3 HTTP/1.0" 202 Accepted 2026-03-29 09:08:47,774 - lava_callback - INFO - Starting async job submit for node 69c8ebdc6a4c026e163489d3 2026-03-29 09:08:47,803 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:08:47,817 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8ebdc6a4c026e163489d3 HTTP/1.1" 200 None 2026-03-29 09:08:47,818 - lava_callback - DEBUG - Processing node 69c8ebdc6a4c026e163489d3: baseline-arm64 2026-03-29 09:08:47,954 - lava_callback - WARNING - Failed to upload log 2026-03-29 09:08:48,023 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:08:48,231 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:08:48,233 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8ebdc6a4c026e163489d3/lava-logs.yaml 2026-03-29 09:08:48,237 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:08:48,385 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:08:48,387 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8ebdc6a4c026e163489d3/lava_callback.json.gz 2026-03-29 09:08:48,418 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:08:48,431 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8ebdc6a4c026e163489d3 HTTP/1.1" 200 None 2026-03-29 09:08:48,433 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:08:48,451 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 09:08:48,453 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:08:48,474 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8ebdc6a4c026e163489d3 HTTP/1.1" 200 None 2026-03-29 09:08:48,558 - lava_callback - INFO - Completed processing callback for node 69c8ebdc6a4c026e163489d3 INFO: 172.21.0.1:49378 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50702 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50704 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:09:09,853 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:09:09,869 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:58012 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:58018 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44342 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:09:31,730 - lava_callback - INFO - Received callback for node 69c64941b98d4d3c02068919 INFO: 172.21.0.1:40496 - "POST /node/69c64941b98d4d3c02068919 HTTP/1.0" 202 Accepted 2026-03-29 09:09:31,736 - lava_callback - INFO - Starting async job submit for node 69c64941b98d4d3c02068919 2026-03-29 09:09:31,737 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:09:31,752 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64941b98d4d3c02068919 HTTP/1.1" 200 None 2026-03-29 09:09:31,753 - lava_callback - DEBUG - Processing node 69c64941b98d4d3c02068919: baseline-arm64-baylibre 2026-03-29 09:09:31,753 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64941b98d4d3c02068919 2026-03-29 09:09:31,754 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:09:31,776 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64941b98d4d3c02068919 HTTP/1.1" 200 None 2026-03-29 09:09:31,778 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:09:31,793 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c399cdbbc022e792643ea0 HTTP/1.1" 200 None 2026-03-29 09:09:31,794 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:09:31,814 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64941b98d4d3c02068919 HTTP/1.1" 200 None 2026-03-29 09:09:31,815 - lava_callback - ERROR - Error processing callback for node 69c64941b98d4d3c02068919: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:40504 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51744 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58102 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54936 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54940 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48532 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48544 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58334 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:10:31,942 - lava_callback - INFO - Received callback for node 69c64983b98d4d3c020689a9 INFO: 172.21.0.1:38610 - "POST /node/69c64983b98d4d3c020689a9 HTTP/1.0" 202 Accepted 2026-03-29 09:10:31,948 - lava_callback - INFO - Starting async job submit for node 69c64983b98d4d3c020689a9 2026-03-29 09:10:31,949 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:31,987 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64983b98d4d3c020689a9 HTTP/1.1" 200 None 2026-03-29 09:10:31,987 - lava_callback - DEBUG - Processing node 69c64983b98d4d3c020689a9: baseline-arm64-baylibre 2026-03-29 09:10:31,988 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64983b98d4d3c020689a9 2026-03-29 09:10:31,988 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:32,002 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64983b98d4d3c020689a9 HTTP/1.1" 200 None 2026-03-29 09:10:32,004 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:32,029 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63cf0b98d4d3c02067dd7 HTTP/1.1" 200 None 2026-03-29 09:10:32,030 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:32,051 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64983b98d4d3c020689a9 HTTP/1.1" 200 None 2026-03-29 09:10:32,057 - lava_callback - ERROR - Error processing callback for node 69c64983b98d4d3c020689a9: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-29 09:10:32,655 - lava_callback - INFO - Received callback for node 69c64986b98d4d3c020689ac INFO: 172.21.0.1:38614 - "POST /node/69c64986b98d4d3c020689ac HTTP/1.0" 202 Accepted 2026-03-29 09:10:32,661 - lava_callback - INFO - Starting async job submit for node 69c64986b98d4d3c020689ac 2026-03-29 09:10:32,662 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:32,676 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64986b98d4d3c020689ac HTTP/1.1" 200 None 2026-03-29 09:10:32,677 - lava_callback - DEBUG - Processing node 69c64986b98d4d3c020689ac: baseline-arm64-baylibre 2026-03-29 09:10:32,677 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64986b98d4d3c020689ac 2026-03-29 09:10:32,678 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:32,692 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64986b98d4d3c020689ac HTTP/1.1" 200 None 2026-03-29 09:10:32,693 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:32,710 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63cf0b98d4d3c02067dd7 HTTP/1.1" 200 None 2026-03-29 09:10:32,711 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:32,733 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64986b98d4d3c020689ac HTTP/1.1" 200 None 2026-03-29 09:10:32,739 - lava_callback - ERROR - Error processing callback for node 69c64986b98d4d3c020689ac: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-29 09:10:33,337 - lava_callback - INFO - Received callback for node 69c64982b98d4d3c020689a6 INFO: 172.21.0.1:38624 - "POST /node/69c64982b98d4d3c020689a6 HTTP/1.0" 202 Accepted 2026-03-29 09:10:33,342 - lava_callback - INFO - Starting async job submit for node 69c64982b98d4d3c020689a6 2026-03-29 09:10:33,343 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:33,366 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64982b98d4d3c020689a6 HTTP/1.1" 200 None 2026-03-29 09:10:33,367 - lava_callback - DEBUG - Processing node 69c64982b98d4d3c020689a6: baseline-arm64-baylibre 2026-03-29 09:10:33,367 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64982b98d4d3c020689a6 2026-03-29 09:10:33,368 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:33,381 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64982b98d4d3c020689a6 HTTP/1.1" 200 None 2026-03-29 09:10:33,382 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:33,396 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63cf0b98d4d3c02067dd7 HTTP/1.1" 200 None 2026-03-29 09:10:33,397 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:33,417 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64982b98d4d3c020689a6 HTTP/1.1" 200 None 2026-03-29 09:10:33,422 - lava_callback - ERROR - Error processing callback for node 69c64982b98d4d3c020689a6: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-29 09:10:34,904 - lava_callback - INFO - Received callback for node 69c8ec656a4c026e163489e8 INFO: 172.21.0.1:38628 - "POST /node/69c8ec656a4c026e163489e8 HTTP/1.0" 202 Accepted 2026-03-29 09:10:34,910 - lava_callback - INFO - Starting async job submit for node 69c8ec656a4c026e163489e8 2026-03-29 09:10:34,928 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:34,942 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8ec656a4c026e163489e8 HTTP/1.1" 200 None 2026-03-29 09:10:34,942 - lava_callback - DEBUG - Processing node 69c8ec656a4c026e163489e8: baseline-x86 2026-03-29 09:10:35,138 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:10:35,207 - lava_callback - INFO - Received callback for node 69c8ec646a4c026e163489e5 INFO: 172.21.0.1:38640 - "POST /node/69c8ec646a4c026e163489e5 HTTP/1.0" 202 Accepted 2026-03-29 09:10:35,214 - lava_callback - INFO - Starting async job submit for node 69c8ec646a4c026e163489e5 2026-03-29 09:10:35,233 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:35,248 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8ec646a4c026e163489e5 HTTP/1.1" 200 None 2026-03-29 09:10:35,249 - lava_callback - DEBUG - Processing node 69c8ec646a4c026e163489e5: baseline-x86 2026-03-29 09:10:35,312 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:10:35,354 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-x86-69c8ec656a4c026e163489e8/log.txt.gz 2026-03-29 09:10:35,530 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:10:35,530 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:10:35,697 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:10:35,699 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-x86-69c8ec646a4c026e163489e5/log.txt.gz 2026-03-29 09:10:35,775 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:10:35,806 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69c8ec656a4c026e163489e8/lava-logs.yaml 2026-03-29 09:10:35,808 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:10:35,812 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:10:35,990 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:10:35,992 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69c8ec656a4c026e163489e8/lava_callback.json.gz 2026-03-29 09:10:36,011 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:36,023 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:10:36,025 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69c8ec646a4c026e163489e5/lava-logs.yaml 2026-03-29 09:10:36,028 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8ec656a4c026e163489e8 HTTP/1.1" 200 None 2026-03-29 09:10:36,030 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:36,031 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:10:36,044 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e00a6a4c026e16347d61 HTTP/1.1" 200 None 2026-03-29 09:10:36,045 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:36,100 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8ec656a4c026e163489e8 HTTP/1.1" 200 None 2026-03-29 09:10:36,103 - lava_callback - INFO - Completed processing callback for node 69c8ec656a4c026e163489e8 2026-03-29 09:10:36,219 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:10:36,221 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69c8ec646a4c026e163489e5/lava_callback.json.gz 2026-03-29 09:10:36,246 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:36,260 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8ec646a4c026e163489e5 HTTP/1.1" 200 None 2026-03-29 09:10:36,261 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:36,276 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e00a6a4c026e16347d61 HTTP/1.1" 200 None 2026-03-29 09:10:36,278 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:36,311 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8ec646a4c026e163489e5 HTTP/1.1" 200 None 2026-03-29 09:10:36,313 - lava_callback - INFO - Completed processing callback for node 69c8ec646a4c026e163489e5 INFO: 172.21.0.1:38656 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:10:39,871 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:39,894 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:60976 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:10:51,760 - lava_callback - INFO - Received callback for node 69c6498bb98d4d3c020689ba INFO: 172.21.0.1:47770 - "POST /node/69c6498bb98d4d3c020689ba HTTP/1.0" 202 Accepted 2026-03-29 09:10:51,766 - lava_callback - INFO - Starting async job submit for node 69c6498bb98d4d3c020689ba 2026-03-29 09:10:51,766 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:51,781 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6498bb98d4d3c020689ba HTTP/1.1" 200 None 2026-03-29 09:10:51,781 - lava_callback - DEBUG - Processing node 69c6498bb98d4d3c020689ba: baseline-arm64-baylibre 2026-03-29 09:10:51,782 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c6498bb98d4d3c020689ba 2026-03-29 09:10:51,782 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:51,820 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6498bb98d4d3c020689ba HTTP/1.1" 200 None 2026-03-29 09:10:51,822 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:51,835 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63cf0b98d4d3c02067dd7 HTTP/1.1" 200 None 2026-03-29 09:10:51,837 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:51,857 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c6498bb98d4d3c020689ba HTTP/1.1" 200 None 2026-03-29 09:10:51,866 - lava_callback - ERROR - Error processing callback for node 69c6498bb98d4d3c020689ba: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-29 09:10:52,445 - lava_callback - INFO - Received callback for node 69c64988b98d4d3c020689b3 INFO: 172.21.0.1:47780 - "POST /node/69c64988b98d4d3c020689b3 HTTP/1.0" 202 Accepted 2026-03-29 09:10:52,451 - lava_callback - INFO - Starting async job submit for node 69c64988b98d4d3c020689b3 2026-03-29 09:10:52,452 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:52,466 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64988b98d4d3c020689b3 HTTP/1.1" 200 None 2026-03-29 09:10:52,467 - lava_callback - DEBUG - Processing node 69c64988b98d4d3c020689b3: baseline-arm64-baylibre 2026-03-29 09:10:52,467 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64988b98d4d3c020689b3 2026-03-29 09:10:52,468 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:52,481 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64988b98d4d3c020689b3 HTTP/1.1" 200 None 2026-03-29 09:10:52,482 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:52,496 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63cf0b98d4d3c02067dd7 HTTP/1.1" 200 None 2026-03-29 09:10:52,497 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:52,522 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64988b98d4d3c020689b3 HTTP/1.1" 200 None 2026-03-29 09:10:52,528 - lava_callback - ERROR - Error processing callback for node 69c64988b98d4d3c020689b3: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-29 09:10:53,130 - lava_callback - INFO - Received callback for node 69c6498ab98d4d3c020689b7 INFO: 172.21.0.1:47786 - "POST /node/69c6498ab98d4d3c020689b7 HTTP/1.0" 202 Accepted 2026-03-29 09:10:53,136 - lava_callback - INFO - Starting async job submit for node 69c6498ab98d4d3c020689b7 2026-03-29 09:10:53,137 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:53,151 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6498ab98d4d3c020689b7 HTTP/1.1" 200 None 2026-03-29 09:10:53,151 - lava_callback - DEBUG - Processing node 69c6498ab98d4d3c020689b7: baseline-arm64-baylibre 2026-03-29 09:10:53,152 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c6498ab98d4d3c020689b7 2026-03-29 09:10:53,152 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:53,165 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6498ab98d4d3c020689b7 HTTP/1.1" 200 None 2026-03-29 09:10:53,167 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:53,180 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63cf0b98d4d3c02067dd7 HTTP/1.1" 200 None 2026-03-29 09:10:53,182 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:53,208 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c6498ab98d4d3c020689b7 HTTP/1.1" 200 None 2026-03-29 09:10:53,209 - lava_callback - ERROR - Error processing callback for node 69c6498ab98d4d3c020689b7: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-29 09:10:53,807 - lava_callback - INFO - Received callback for node 69c64990b98d4d3c020689c3 INFO: 172.21.0.1:47790 - "POST /node/69c64990b98d4d3c020689c3 HTTP/1.0" 202 Accepted 2026-03-29 09:10:53,812 - lava_callback - INFO - Starting async job submit for node 69c64990b98d4d3c020689c3 2026-03-29 09:10:53,814 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:53,827 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64990b98d4d3c020689c3 HTTP/1.1" 200 None 2026-03-29 09:10:53,828 - lava_callback - DEBUG - Processing node 69c64990b98d4d3c020689c3: baseline-arm64-baylibre 2026-03-29 09:10:53,828 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64990b98d4d3c020689c3 2026-03-29 09:10:53,829 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:53,842 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64990b98d4d3c020689c3 HTTP/1.1" 200 None 2026-03-29 09:10:53,843 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:53,856 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c399cdbbc022e792643ea0 HTTP/1.1" 200 None 2026-03-29 09:10:53,858 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:10:53,883 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64990b98d4d3c020689c3 HTTP/1.1" 200 None 2026-03-29 09:10:53,885 - lava_callback - ERROR - Error processing callback for node 69c64990b98d4d3c020689c3: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:47792 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42720 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:11:05,164 - lava_callback - INFO - Received callback for node 69c6499ab98d4d3c020689f6 INFO: 172.21.0.1:42732 - "POST /node/69c6499ab98d4d3c020689f6 HTTP/1.0" 202 Accepted 2026-03-29 09:11:05,170 - lava_callback - INFO - Starting async job submit for node 69c6499ab98d4d3c020689f6 2026-03-29 09:11:05,170 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:05,195 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6499ab98d4d3c020689f6 HTTP/1.1" 200 None 2026-03-29 09:11:05,196 - lava_callback - DEBUG - Processing node 69c6499ab98d4d3c020689f6: baseline-arm64-clabbe 2026-03-29 09:11:05,196 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c6499ab98d4d3c020689f6 2026-03-29 09:11:05,197 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:05,210 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6499ab98d4d3c020689f6 HTTP/1.1" 200 None 2026-03-29 09:11:05,212 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:05,225 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63cf0b98d4d3c02067dd7 HTTP/1.1" 200 None 2026-03-29 09:11:05,226 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:05,256 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c6499ab98d4d3c020689f6 HTTP/1.1" 200 None 2026-03-29 09:11:05,264 - lava_callback - ERROR - Error processing callback for node 69c6499ab98d4d3c020689f6: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-29 09:11:06,074 - lava_callback - INFO - Received callback for node 69c6499cb98d4d3c020689f9 INFO: 172.21.0.1:42734 - "POST /node/69c6499cb98d4d3c020689f9 HTTP/1.0" 202 Accepted 2026-03-29 09:11:06,080 - lava_callback - INFO - Starting async job submit for node 69c6499cb98d4d3c020689f9 2026-03-29 09:11:06,080 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:06,098 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6499cb98d4d3c020689f9 HTTP/1.1" 200 None 2026-03-29 09:11:06,099 - lava_callback - DEBUG - Processing node 69c6499cb98d4d3c020689f9: baseline-arm64-clabbe 2026-03-29 09:11:06,099 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c6499cb98d4d3c020689f9 2026-03-29 09:11:06,100 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:06,113 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6499cb98d4d3c020689f9 HTTP/1.1" 200 None 2026-03-29 09:11:06,114 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:06,138 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63cf0b98d4d3c02067dd7 HTTP/1.1" 200 None 2026-03-29 09:11:06,139 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:06,161 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c6499cb98d4d3c020689f9 HTTP/1.1" 200 None 2026-03-29 09:11:06,170 - lava_callback - ERROR - Error processing callback for node 69c6499cb98d4d3c020689f9: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:42738 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:11:06,784 - lava_callback - INFO - Received callback for node 69c6499db98d4d3c02068a02 INFO: 172.21.0.1:42748 - "POST /node/69c6499db98d4d3c02068a02 HTTP/1.0" 202 Accepted 2026-03-29 09:11:06,789 - lava_callback - INFO - Starting async job submit for node 69c6499db98d4d3c02068a02 2026-03-29 09:11:06,790 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:06,804 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6499db98d4d3c02068a02 HTTP/1.1" 200 None 2026-03-29 09:11:06,805 - lava_callback - DEBUG - Processing node 69c6499db98d4d3c02068a02: baseline-arm64-clabbe 2026-03-29 09:11:06,805 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c6499db98d4d3c02068a02 2026-03-29 09:11:06,806 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:06,823 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6499db98d4d3c02068a02 HTTP/1.1" 200 None 2026-03-29 09:11:06,825 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:06,838 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63cf0b98d4d3c02067dd7 HTTP/1.1" 200 None 2026-03-29 09:11:06,840 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:06,860 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c6499db98d4d3c02068a02 HTTP/1.1" 200 None 2026-03-29 09:11:06,866 - lava_callback - ERROR - Error processing callback for node 69c6499db98d4d3c02068a02: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-29 09:11:07,490 - lava_callback - INFO - Received callback for node 69c6499eb98d4d3c02068a05 INFO: 172.21.0.1:42762 - "POST /node/69c6499eb98d4d3c02068a05 HTTP/1.0" 202 Accepted 2026-03-29 09:11:07,495 - lava_callback - INFO - Starting async job submit for node 69c6499eb98d4d3c02068a05 2026-03-29 09:11:07,496 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:07,521 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6499eb98d4d3c02068a05 HTTP/1.1" 200 None 2026-03-29 09:11:07,522 - lava_callback - DEBUG - Processing node 69c6499eb98d4d3c02068a05: baseline-arm64-clabbe 2026-03-29 09:11:07,522 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c6499eb98d4d3c02068a05 2026-03-29 09:11:07,523 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:07,536 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6499eb98d4d3c02068a05 HTTP/1.1" 200 None 2026-03-29 09:11:07,537 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:07,551 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63cf0b98d4d3c02067dd7 HTTP/1.1" 200 None 2026-03-29 09:11:07,553 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:07,573 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c6499eb98d4d3c02068a05 HTTP/1.1" 200 None 2026-03-29 09:11:07,583 - lava_callback - ERROR - Error processing callback for node 69c6499eb98d4d3c02068a05: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-29 09:11:08,185 - lava_callback - INFO - Received callback for node 69c649a0b98d4d3c02068a0b INFO: 172.21.0.1:42776 - "POST /node/69c649a0b98d4d3c02068a0b HTTP/1.0" 202 Accepted 2026-03-29 09:11:08,190 - lava_callback - INFO - Starting async job submit for node 69c649a0b98d4d3c02068a0b 2026-03-29 09:11:08,191 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:08,213 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c649a0b98d4d3c02068a0b HTTP/1.1" 200 None 2026-03-29 09:11:08,214 - lava_callback - DEBUG - Processing node 69c649a0b98d4d3c02068a0b: baseline-arm64-clabbe 2026-03-29 09:11:08,214 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c649a0b98d4d3c02068a0b 2026-03-29 09:11:08,215 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:08,229 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c649a0b98d4d3c02068a0b HTTP/1.1" 200 None 2026-03-29 09:11:08,230 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:08,247 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63cf0b98d4d3c02067dd7 HTTP/1.1" 200 None 2026-03-29 09:11:08,249 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:08,276 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c649a0b98d4d3c02068a0b HTTP/1.1" 200 None 2026-03-29 09:11:08,277 - lava_callback - ERROR - Error processing callback for node 69c649a0b98d4d3c02068a0b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-29 09:11:08,885 - lava_callback - INFO - Received callback for node 69c649a1b98d4d3c02068a0e INFO: 172.21.0.1:42786 - "POST /node/69c649a1b98d4d3c02068a0e HTTP/1.0" 202 Accepted 2026-03-29 09:11:08,890 - lava_callback - INFO - Starting async job submit for node 69c649a1b98d4d3c02068a0e 2026-03-29 09:11:08,891 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:08,906 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c649a1b98d4d3c02068a0e HTTP/1.1" 200 None 2026-03-29 09:11:08,907 - lava_callback - DEBUG - Processing node 69c649a1b98d4d3c02068a0e: baseline-arm64-clabbe 2026-03-29 09:11:08,907 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c649a1b98d4d3c02068a0e 2026-03-29 09:11:08,907 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:08,924 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c649a1b98d4d3c02068a0e HTTP/1.1" 200 None 2026-03-29 09:11:08,925 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:08,938 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63cf0b98d4d3c02067dd7 HTTP/1.1" 200 None 2026-03-29 09:11:08,940 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:08,965 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c649a1b98d4d3c02068a0e HTTP/1.1" 200 None 2026-03-29 09:11:08,967 - lava_callback - ERROR - Error processing callback for node 69c649a1b98d4d3c02068a0e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:52692 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:52706 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43000 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:11:32,834 - lava_callback - INFO - Received callback for node 69c8e8216a4c026e1634870c INFO: 172.21.0.1:54062 - "POST /node/69c8e8216a4c026e1634870c HTTP/1.0" 202 Accepted 2026-03-29 09:11:32,844 - lava_callback - INFO - Starting async job submit for node 69c8e8216a4c026e1634870c 2026-03-29 09:11:32,937 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:32,951 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8216a4c026e1634870c HTTP/1.1" 200 None 2026-03-29 09:11:32,952 - lava_callback - DEBUG - Processing node 69c8e8216a4c026e1634870c: baseline-nfs-arm64 2026-03-29 09:11:33,995 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:11:34,143 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:11:34,145 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e8216a4c026e1634870c/log.txt.gz 2026-03-29 09:11:34,731 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:11:34,944 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:11:34,947 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e8216a4c026e1634870c/lava-logs.yaml 2026-03-29 09:11:34,966 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:11:35,167 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:11:35,169 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e8216a4c026e1634870c/lava_callback.json.gz DEBUG: Setup failed for 69c8e8216a4c026e1634870c. Transitting job node result: incomplete -> fail 2026-03-29 09:11:35,258 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:35,272 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8216a4c026e1634870c HTTP/1.1" 200 None 2026-03-29 09:11:35,274 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:35,296 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 09:11:35,297 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:35,332 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8216a4c026e1634870c HTTP/1.1" 200 None 2026-03-29 09:11:35,435 - lava_callback - INFO - Completed processing callback for node 69c8e8216a4c026e1634870c INFO: 172.21.0.1:54074 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:11:39,896 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:39,912 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-29 09:11:40,302 - lava_callback - INFO - Received callback for node 69c8ec8f6a4c026e16348a08 INFO: 172.21.0.1:35030 - "POST /node/69c8ec8f6a4c026e16348a08 HTTP/1.0" 202 Accepted 2026-03-29 09:11:40,308 - lava_callback - INFO - Starting async job submit for node 69c8ec8f6a4c026e16348a08 2026-03-29 09:11:40,338 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:40,351 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8ec8f6a4c026e16348a08 HTTP/1.1" 200 None 2026-03-29 09:11:40,352 - lava_callback - DEBUG - Processing node 69c8ec8f6a4c026e16348a08: baseline-arm64 2026-03-29 09:11:40,486 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:11:40,625 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:11:40,627 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8ec8f6a4c026e16348a08/log.txt.gz 2026-03-29 09:11:40,705 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:11:40,910 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:11:40,912 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8ec8f6a4c026e16348a08/lava-logs.yaml 2026-03-29 09:11:40,916 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:11:41,042 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:11:41,044 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8ec8f6a4c026e16348a08/lava_callback.json.gz 2026-03-29 09:11:41,076 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:41,091 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8ec8f6a4c026e16348a08 HTTP/1.1" 200 None 2026-03-29 09:11:41,092 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:41,105 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 09:11:41,107 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:11:41,129 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8ec8f6a4c026e16348a08 HTTP/1.1" 200 None 2026-03-29 09:11:41,205 - lava_callback - INFO - Completed processing callback for node 69c8ec8f6a4c026e16348a08 INFO: 172.21.0.1:35038 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55876 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44552 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44564 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:12:09,914 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:12:09,933 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-29 09:12:11,838 - lava_callback - INFO - Received callback for node 69c649dfb98d4d3c02068a78 INFO: 172.21.0.1:36456 - "POST /node/69c649dfb98d4d3c02068a78 HTTP/1.0" 202 Accepted 2026-03-29 09:12:11,844 - lava_callback - INFO - Starting async job submit for node 69c649dfb98d4d3c02068a78 2026-03-29 09:12:11,845 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:12:11,860 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c649dfb98d4d3c02068a78 HTTP/1.1" 200 None 2026-03-29 09:12:11,861 - lava_callback - DEBUG - Processing node 69c649dfb98d4d3c02068a78: baseline-arm64-baylibre 2026-03-29 09:12:11,861 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c649dfb98d4d3c02068a78 2026-03-29 09:12:11,862 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:12:11,876 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c649dfb98d4d3c02068a78 HTTP/1.1" 200 None 2026-03-29 09:12:11,877 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:12:11,898 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c399cdbbc022e792643ea0 HTTP/1.1" 200 None 2026-03-29 09:12:11,899 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:12:11,921 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c649dfb98d4d3c02068a78 HTTP/1.1" 200 None 2026-03-29 09:12:11,922 - lava_callback - ERROR - Error processing callback for node 69c649dfb98d4d3c02068a78: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:36468 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:36484 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:12:20,218 - lava_callback - INFO - Received callback for node 69c8ecdc6a4c026e16348a31 INFO: 172.21.0.1:53772 - "POST /node/69c8ecdc6a4c026e16348a31 HTTP/1.0" 202 Accepted 2026-03-29 09:12:20,224 - lava_callback - INFO - Starting async job submit for node 69c8ecdc6a4c026e16348a31 2026-03-29 09:12:20,227 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:12:20,242 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8ecdc6a4c026e16348a31 HTTP/1.1" 200 None 2026-03-29 09:12:20,243 - lava_callback - DEBUG - Processing node 69c8ecdc6a4c026e16348a31: baseline-arm 2026-03-29 09:12:20,248 - lava_callback - WARNING - Failed to upload log 2026-03-29 09:12:20,251 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:12:20,414 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:12:20,416 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69c8ecdc6a4c026e16348a31/lava-logs.yaml 2026-03-29 09:12:20,418 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:12:20,564 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:12:20,566 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69c8ecdc6a4c026e16348a31/lava_callback.json.gz 2026-03-29 09:12:20,569 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:12:20,583 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8ecdc6a4c026e16348a31 HTTP/1.1" 200 None 2026-03-29 09:12:20,584 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:12:20,597 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 09:12:20,599 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:12:20,619 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8ecdc6a4c026e16348a31 HTTP/1.1" 200 None 2026-03-29 09:12:20,629 - lava_callback - INFO - Completed processing callback for node 69c8ecdc6a4c026e16348a31 INFO: 172.21.0.1:53776 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35040 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:12:39,935 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:12:39,952 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:49800 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45646 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60212 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60222 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38728 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:38730 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57638 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:13:31,931 - lava_callback - INFO - Received callback for node 69c64a2cb98d4d3c02068b09 INFO: 172.21.0.1:51154 - "POST /node/69c64a2cb98d4d3c02068b09 HTTP/1.0" 202 Accepted 2026-03-29 09:13:31,937 - lava_callback - INFO - Starting async job submit for node 69c64a2cb98d4d3c02068b09 2026-03-29 09:13:31,938 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:13:31,952 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64a2cb98d4d3c02068b09 HTTP/1.1" 200 None 2026-03-29 09:13:31,953 - lava_callback - DEBUG - Processing node 69c64a2cb98d4d3c02068b09: baseline-arm64-baylibre 2026-03-29 09:13:31,953 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64a2cb98d4d3c02068b09 2026-03-29 09:13:31,954 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:13:31,967 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64a2cb98d4d3c02068b09 HTTP/1.1" 200 None 2026-03-29 09:13:31,968 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:13:31,981 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c399cdbbc022e792643ea0 HTTP/1.1" 200 None 2026-03-29 09:13:31,983 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:13:32,004 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64a2cb98d4d3c02068b09 HTTP/1.1" 200 None 2026-03-29 09:13:32,015 - lava_callback - ERROR - Error processing callback for node 69c64a2cb98d4d3c02068b09: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:51168 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43240 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35660 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36130 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36132 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49658 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:49660 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49908 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:14:31,700 - lava_callback - INFO - Received callback for node 69c8ed036a4c026e16348a41 INFO: 172.21.0.1:41834 - "POST /node/69c8ed036a4c026e16348a41 HTTP/1.0" 202 Accepted 2026-03-29 09:14:31,710 - lava_callback - INFO - Starting async job submit for node 69c8ed036a4c026e16348a41 2026-03-29 09:14:31,734 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:14:31,749 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8ed036a4c026e16348a41 HTTP/1.1" 200 None 2026-03-29 09:14:31,750 - lava_callback - DEBUG - Processing node 69c8ed036a4c026e16348a41: baseline-arm64 2026-03-29 09:14:32,224 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:14:32,368 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:14:32,370 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8ed036a4c026e16348a41/log.txt.gz 2026-03-29 09:14:32,638 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:14:32,960 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:14:32,962 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8ed036a4c026e16348a41/lava-logs.yaml 2026-03-29 09:14:32,971 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:14:33,133 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:14:33,135 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8ed036a4c026e16348a41/lava_callback.json.gz 2026-03-29 09:14:33,182 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:14:33,211 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8ed036a4c026e16348a41 HTTP/1.1" 200 None 2026-03-29 09:14:33,214 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:14:33,237 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 09:14:33,239 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:14:33,275 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8ed036a4c026e16348a41 HTTP/1.1" 200 None 2026-03-29 09:14:33,338 - lava_callback - INFO - Completed processing callback for node 69c8ed036a4c026e16348a41 INFO: 172.21.0.1:41838 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:14:39,954 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:14:39,973 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-29 09:14:45,325 - lava_callback - INFO - Received callback for node 69c64a7fb98d4d3c02068b23 INFO: 172.21.0.1:55270 - "POST /node/69c64a7fb98d4d3c02068b23 HTTP/1.0" 202 Accepted 2026-03-29 09:14:45,330 - lava_callback - INFO - Starting async job submit for node 69c64a7fb98d4d3c02068b23 2026-03-29 09:14:45,331 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:14:45,347 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64a7fb98d4d3c02068b23 HTTP/1.1" 200 None 2026-03-29 09:14:45,347 - lava_callback - DEBUG - Processing node 69c64a7fb98d4d3c02068b23: baseline-arm64-clabbe 2026-03-29 09:14:45,348 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64a7fb98d4d3c02068b23 2026-03-29 09:14:45,348 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:14:45,362 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64a7fb98d4d3c02068b23 HTTP/1.1" 200 None 2026-03-29 09:14:45,363 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:14:45,377 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c399cdbbc022e792643ea0 HTTP/1.1" 200 None 2026-03-29 09:14:45,378 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:14:45,405 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64a7fb98d4d3c02068b23 HTTP/1.1" 200 None 2026-03-29 09:14:45,406 - lava_callback - ERROR - Error processing callback for node 69c64a7fb98d4d3c02068b23: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:55274 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54330 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59596 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59610 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39640 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:39642 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49084 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45022 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40914 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41474 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49558 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:16:05,215 - lava_callback - INFO - Received callback for node 69c64acbb98d4d3c02068b38 INFO: 172.21.0.1:49568 - "POST /node/69c64acbb98d4d3c02068b38 HTTP/1.0" 202 Accepted 2026-03-29 09:16:05,221 - lava_callback - INFO - Starting async job submit for node 69c64acbb98d4d3c02068b38 2026-03-29 09:16:05,222 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:16:05,245 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64acbb98d4d3c02068b38 HTTP/1.1" 200 None 2026-03-29 09:16:05,245 - lava_callback - DEBUG - Processing node 69c64acbb98d4d3c02068b38: baseline-arm64-clabbe 2026-03-29 09:16:05,246 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64acbb98d4d3c02068b38 2026-03-29 09:16:05,246 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:16:05,260 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64acbb98d4d3c02068b38 HTTP/1.1" 200 None 2026-03-29 09:16:05,261 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:16:05,274 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c399cdbbc022e792643ea0 HTTP/1.1" 200 None 2026-03-29 09:16:05,276 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:16:05,298 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64acbb98d4d3c02068b38 HTTP/1.1" 200 None 2026-03-29 09:16:05,300 - lava_callback - ERROR - Error processing callback for node 69c64acbb98d4d3c02068b38: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:49572 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51670 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:51680 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34238 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46370 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36568 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57492 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32936 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32948 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53366 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:53368 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:17:25,333 - lava_callback - INFO - Received callback for node 69c64b19b98d4d3c02068b4e INFO: 172.21.0.1:59932 - "POST /node/69c64b19b98d4d3c02068b4e HTTP/1.0" 202 Accepted 2026-03-29 09:17:25,339 - lava_callback - INFO - Starting async job submit for node 69c64b19b98d4d3c02068b4e 2026-03-29 09:17:25,339 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:17:25,354 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64b19b98d4d3c02068b4e HTTP/1.1" 200 None 2026-03-29 09:17:25,355 - lava_callback - DEBUG - Processing node 69c64b19b98d4d3c02068b4e: baseline-arm64-clabbe 2026-03-29 09:17:25,355 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64b19b98d4d3c02068b4e 2026-03-29 09:17:25,356 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:17:25,370 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64b19b98d4d3c02068b4e HTTP/1.1" 200 None 2026-03-29 09:17:25,371 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:17:25,385 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c399cdbbc022e792643ea0 HTTP/1.1" 200 None 2026-03-29 09:17:25,386 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:17:25,410 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64b19b98d4d3c02068b4e HTTP/1.1" 200 None 2026-03-29 09:17:25,419 - lava_callback - ERROR - Error processing callback for node 69c64b19b98d4d3c02068b4e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:59942 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46580 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44360 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49770 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40398 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40414 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44254 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:44260 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42496 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60920 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:18:45,435 - lava_callback - INFO - Received callback for node 69c64b66b98d4d3c02068b62 INFO: 172.21.0.1:35894 - "POST /node/69c64b66b98d4d3c02068b62 HTTP/1.0" 202 Accepted 2026-03-29 09:18:45,440 - lava_callback - INFO - Starting async job submit for node 69c64b66b98d4d3c02068b62 2026-03-29 09:18:45,441 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:18:45,455 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64b66b98d4d3c02068b62 HTTP/1.1" 200 None 2026-03-29 09:18:45,456 - lava_callback - DEBUG - Processing node 69c64b66b98d4d3c02068b62: baseline-arm64-clabbe 2026-03-29 09:18:45,456 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64b66b98d4d3c02068b62 2026-03-29 09:18:45,457 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:18:45,471 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64b66b98d4d3c02068b62 HTTP/1.1" 200 None 2026-03-29 09:18:45,472 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:18:45,486 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c399cdbbc022e792643ea0 HTTP/1.1" 200 None 2026-03-29 09:18:45,488 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:18:45,511 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64b66b98d4d3c02068b62 HTTP/1.1" 200 None 2026-03-29 09:18:45,518 - lava_callback - ERROR - Error processing callback for node 69c64b66b98d4d3c02068b62: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:35908 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51964 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:19:02,220 - lava_callback - INFO - Received callback for node 69c8ea676a4c026e16348988 INFO: 172.21.0.1:36904 - "POST /node/69c8ea676a4c026e16348988 HTTP/1.0" 202 Accepted 2026-03-29 09:19:02,227 - lava_callback - INFO - Starting async job submit for node 69c8ea676a4c026e16348988 2026-03-29 09:19:02,287 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:19:02,302 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8ea676a4c026e16348988 HTTP/1.1" 200 None 2026-03-29 09:19:02,303 - lava_callback - DEBUG - Processing node 69c8ea676a4c026e16348988: ltp-smoke 2026-03-29 09:19:02,801 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:19:02,966 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:19:02,968 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-smoke-69c8ea676a4c026e16348988/log.txt.gz 2026-03-29 09:19:03,261 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:19:03,485 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:19:03,487 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-smoke-69c8ea676a4c026e16348988/lava-logs.yaml 2026-03-29 09:19:03,498 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:19:03,687 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:19:03,689 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-smoke-69c8ea676a4c026e16348988/lava_callback.json.gz 2026-03-29 09:19:03,822 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:19:03,836 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8ea676a4c026e16348988 HTTP/1.1" 200 None 2026-03-29 09:19:03,838 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:19:03,851 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff06a4c026e16347d3c HTTP/1.1" 200 None 2026-03-29 09:19:03,853 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:19:03,880 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8ea676a4c026e16348988 HTTP/1.1" 200 None 2026-03-29 09:19:04,021 - lava_callback - INFO - Completed processing callback for node 69c8ea676a4c026e16348988 INFO: 172.21.0.1:36908 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36916 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:19:09,976 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:19:09,993 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:59986 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:59988 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52552 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51870 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46758 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50808 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59000 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:20:05,660 - lava_callback - INFO - Received callback for node 69c64bb3b98d4d3c02068b72 INFO: 172.21.0.1:59004 - "POST /node/69c64bb3b98d4d3c02068b72 HTTP/1.0" 202 Accepted 2026-03-29 09:20:05,666 - lava_callback - INFO - Starting async job submit for node 69c64bb3b98d4d3c02068b72 2026-03-29 09:20:05,667 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:20:05,683 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64bb3b98d4d3c02068b72 HTTP/1.1" 200 None 2026-03-29 09:20:05,684 - lava_callback - DEBUG - Processing node 69c64bb3b98d4d3c02068b72: baseline-arm64-clabbe 2026-03-29 09:20:05,684 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64bb3b98d4d3c02068b72 2026-03-29 09:20:05,685 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:20:05,699 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64bb3b98d4d3c02068b72 HTTP/1.1" 200 None 2026-03-29 09:20:05,700 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:20:05,714 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c399cdbbc022e792643ea0 HTTP/1.1" 200 None 2026-03-29 09:20:05,715 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:20:05,746 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64bb3b98d4d3c02068b72 HTTP/1.1" 200 None 2026-03-29 09:20:05,748 - lava_callback - ERROR - Error processing callback for node 69c64bb3b98d4d3c02068b72: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:59016 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50880 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:50896 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39688 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44764 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56406 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60490 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45172 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45186 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34160 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:34176 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:21:25,529 - lava_callback - INFO - Received callback for node 69c64c02b98d4d3c02068b78 INFO: 172.21.0.1:49842 - "POST /node/69c64c02b98d4d3c02068b78 HTTP/1.0" 202 Accepted 2026-03-29 09:21:25,534 - lava_callback - INFO - Starting async job submit for node 69c64c02b98d4d3c02068b78 2026-03-29 09:21:25,536 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:21:25,556 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64c02b98d4d3c02068b78 HTTP/1.1" 200 None 2026-03-29 09:21:25,557 - lava_callback - DEBUG - Processing node 69c64c02b98d4d3c02068b78: baseline-arm64-clabbe 2026-03-29 09:21:25,557 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64c02b98d4d3c02068b78 2026-03-29 09:21:25,558 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:21:25,571 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64c02b98d4d3c02068b78 HTTP/1.1" 200 None 2026-03-29 09:21:25,573 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:21:25,585 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c399cdbbc022e792643ea0 HTTP/1.1" 200 None 2026-03-29 09:21:25,587 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:21:25,608 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64c02b98d4d3c02068b78 HTTP/1.1" 200 None 2026-03-29 09:21:25,613 - lava_callback - ERROR - Error processing callback for node 69c64c02b98d4d3c02068b78: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:49854 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45902 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:21:36,807 - lava_callback - INFO - Received callback for node 69c8ef126a4c026e16348ac4 INFO: 172.21.0.1:45910 - "POST /node/69c8ef126a4c026e16348ac4 HTTP/1.0" 202 Accepted 2026-03-29 09:21:36,813 - lava_callback - INFO - Starting async job submit for node 69c8ef126a4c026e16348ac4 2026-03-29 09:21:36,815 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:21:36,842 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8ef126a4c026e16348ac4 HTTP/1.1" 200 None 2026-03-29 09:21:36,843 - lava_callback - DEBUG - Processing node 69c8ef126a4c026e16348ac4: baseline-nfs-arm64 2026-03-29 09:21:36,844 - lava_callback - WARNING - Failed to upload log 2026-03-29 09:21:36,845 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:21:36,985 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:21:36,987 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8ef126a4c026e16348ac4/lava-logs.yaml 2026-03-29 09:21:36,988 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:21:37,108 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:21:37,110 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8ef126a4c026e16348ac4/lava_callback.json.gz 2026-03-29 09:21:37,112 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:21:37,128 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8ef126a4c026e16348ac4 HTTP/1.1" 200 None 2026-03-29 09:21:37,130 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:21:37,149 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 09:21:37,152 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:21:37,172 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8ef126a4c026e16348ac4 HTTP/1.1" 200 None 2026-03-29 09:21:37,176 - lava_callback - INFO - Completed processing callback for node 69c8ef126a4c026e16348ac4 2026-03-29 09:21:39,995 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:21:40,014 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:40440 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43514 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39824 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39834 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52872 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:52880 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48588 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:22:32,832 - lava_callback - INFO - Received callback for node 69c8e8176a4c026e163486f0 INFO: 172.21.0.1:36454 - "POST /node/69c8e8176a4c026e163486f0 HTTP/1.0" 202 Accepted 2026-03-29 09:22:32,839 - lava_callback - INFO - Starting async job submit for node 69c8e8176a4c026e163486f0 2026-03-29 09:22:32,864 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:22:32,878 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8176a4c026e163486f0 HTTP/1.1" 200 None 2026-03-29 09:22:32,879 - lava_callback - DEBUG - Processing node 69c8e8176a4c026e163486f0: baseline-arm64 2026-03-29 09:22:33,420 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:22:33,563 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:22:33,565 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8176a4c026e163486f0/log.txt.gz 2026-03-29 09:22:33,825 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:22:34,058 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:22:34,060 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8176a4c026e163486f0/lava-logs.yaml 2026-03-29 09:22:34,073 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:22:34,239 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:22:34,241 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8e8176a4c026e163486f0/lava_callback.json.gz 2026-03-29 09:22:34,267 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:22:34,281 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8176a4c026e163486f0 HTTP/1.1" 200 None 2026-03-29 09:22:34,283 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:22:34,296 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 09:22:34,298 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:22:34,321 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8176a4c026e163486f0 HTTP/1.1" 200 None 2026-03-29 09:22:34,390 - lava_callback - INFO - Completed processing callback for node 69c8e8176a4c026e163486f0 INFO: 172.21.0.1:36458 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:22:40,016 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:22:40,033 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:46896 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45080 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56130 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:23:04,846 - lava_callback - INFO - Received callback for node 69c8ee2c6a4c026e16348a9b INFO: 172.21.0.1:56134 - "POST /node/69c8ee2c6a4c026e16348a9b HTTP/1.0" 202 Accepted 2026-03-29 09:23:04,853 - lava_callback - INFO - Starting async job submit for node 69c8ee2c6a4c026e16348a9b 2026-03-29 09:23:04,890 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:23:04,915 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8ee2c6a4c026e16348a9b HTTP/1.1" 200 None 2026-03-29 09:23:04,916 - lava_callback - DEBUG - Processing node 69c8ee2c6a4c026e16348a9b: baseline-arm64 2026-03-29 09:23:05,320 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:23:05,625 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:23:05,627 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8ee2c6a4c026e16348a9b/log.txt.gz 2026-03-29 09:23:05,841 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:23:06,065 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:23:06,068 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8ee2c6a4c026e16348a9b/lava-logs.yaml 2026-03-29 09:23:06,076 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:23:06,249 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:23:06,251 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8ee2c6a4c026e16348a9b/lava_callback.json.gz DEBUG: Setup failed for 69c8ee2c6a4c026e16348a9b. Transitting job node result: incomplete -> fail 2026-03-29 09:23:06,339 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:23:06,354 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8ee2c6a4c026e16348a9b HTTP/1.1" 200 None 2026-03-29 09:23:06,355 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:23:06,375 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 09:23:06,376 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:23:06,400 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8ee2c6a4c026e16348a9b HTTP/1.1" 200 None 2026-03-29 09:23:06,498 - lava_callback - INFO - Completed processing callback for node 69c8ee2c6a4c026e16348a9b INFO: 172.21.0.1:56136 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:23:10,034 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:23:10,053 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:60748 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:60752 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39048 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35252 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:23:45,576 - lava_callback - INFO - Received callback for node 69c64c8ab98d4d3c02068b9d INFO: 172.21.0.1:47708 - "POST /node/69c64c8ab98d4d3c02068b9d HTTP/1.0" 202 Accepted 2026-03-29 09:23:45,582 - lava_callback - INFO - Starting async job submit for node 69c64c8ab98d4d3c02068b9d 2026-03-29 09:23:45,583 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:23:45,597 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64c8ab98d4d3c02068b9d HTTP/1.1" 200 None 2026-03-29 09:23:45,598 - lava_callback - DEBUG - Processing node 69c64c8ab98d4d3c02068b9d: baseline-arm-clabbe 2026-03-29 09:23:45,598 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64c8ab98d4d3c02068b9d 2026-03-29 09:23:45,599 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:23:45,617 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64c8ab98d4d3c02068b9d HTTP/1.1" 200 None 2026-03-29 09:23:45,619 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:23:45,636 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69be544f448e074b3ffe162d HTTP/1.1" 200 None 2026-03-29 09:23:45,638 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:23:45,669 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64c8ab98d4d3c02068b9d HTTP/1.1" 200 None 2026-03-29 09:23:45,679 - lava_callback - ERROR - Error processing callback for node 69c64c8ab98d4d3c02068b9d: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:47724 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49134 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38652 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38668 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42650 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:42662 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58480 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35900 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:24:45,827 - lava_callback - INFO - Received callback for node 69c64cd4b98d4d3c02068bb0 INFO: 172.21.0.1:47158 - "POST /node/69c64cd4b98d4d3c02068bb0 HTTP/1.0" 202 Accepted 2026-03-29 09:24:45,833 - lava_callback - INFO - Starting async job submit for node 69c64cd4b98d4d3c02068bb0 2026-03-29 09:24:45,834 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:24:45,850 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64cd4b98d4d3c02068bb0 HTTP/1.1" 200 None 2026-03-29 09:24:45,850 - lava_callback - DEBUG - Processing node 69c64cd4b98d4d3c02068bb0: baseline-arm-clabbe 2026-03-29 09:24:45,851 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64cd4b98d4d3c02068bb0 2026-03-29 09:24:45,851 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:24:45,865 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64cd4b98d4d3c02068bb0 HTTP/1.1" 200 None 2026-03-29 09:24:45,866 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:24:45,879 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69be544f448e074b3ffe162d HTTP/1.1" 200 None 2026-03-29 09:24:45,880 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:24:45,901 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64cd4b98d4d3c02068bb0 HTTP/1.1" 200 None 2026-03-29 09:24:45,911 - lava_callback - ERROR - Error processing callback for node 69c64cd4b98d4d3c02068bb0: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:47162 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44888 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57810 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57826 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59878 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:59882 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47112 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48430 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49876 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37642 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58728 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:26:05,690 - lava_callback - INFO - Received callback for node 69c64d1eb98d4d3c02068bc6 INFO: 172.21.0.1:58732 - "POST /node/69c64d1eb98d4d3c02068bc6 HTTP/1.0" 202 Accepted 2026-03-29 09:26:05,696 - lava_callback - INFO - Starting async job submit for node 69c64d1eb98d4d3c02068bc6 2026-03-29 09:26:05,697 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:26:05,714 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64d1eb98d4d3c02068bc6 HTTP/1.1" 200 None 2026-03-29 09:26:05,715 - lava_callback - DEBUG - Processing node 69c64d1eb98d4d3c02068bc6: baseline-arm-clabbe 2026-03-29 09:26:05,715 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64d1eb98d4d3c02068bc6 2026-03-29 09:26:05,715 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:26:05,729 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64d1eb98d4d3c02068bc6 HTTP/1.1" 200 None 2026-03-29 09:26:05,730 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:26:05,753 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69be544f448e074b3ffe162d HTTP/1.1" 200 None 2026-03-29 09:26:05,754 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:26:05,775 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64d1eb98d4d3c02068bc6 HTTP/1.1" 200 None 2026-03-29 09:26:05,776 - lava_callback - ERROR - Error processing callback for node 69c64d1eb98d4d3c02068bc6: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:58740 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50174 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:50178 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60102 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:26:32,757 - lava_callback - INFO - Received callback for node 69c64d3db98d4d3c02068bce INFO: 172.21.0.1:52582 - "POST /node/69c64d3db98d4d3c02068bce HTTP/1.0" 202 Accepted 2026-03-29 09:26:32,763 - lava_callback - INFO - Starting async job submit for node 69c64d3db98d4d3c02068bce 2026-03-29 09:26:32,764 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:26:32,777 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64d3db98d4d3c02068bce HTTP/1.1" 200 None 2026-03-29 09:26:32,778 - lava_callback - DEBUG - Processing node 69c64d3db98d4d3c02068bce: baseline-arm-baylibre 2026-03-29 09:26:32,779 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64d3db98d4d3c02068bce 2026-03-29 09:26:32,779 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:26:32,792 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64d3db98d4d3c02068bce HTTP/1.1" 200 None 2026-03-29 09:26:32,794 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:26:32,811 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c0f72facdb2e26c83cca20 HTTP/1.1" 200 None 2026-03-29 09:26:32,812 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:26:32,833 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64d3db98d4d3c02068bce HTTP/1.1" 200 None 2026-03-29 09:26:32,839 - lava_callback - ERROR - Error processing callback for node 69c64d3db98d4d3c02068bce: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:52598 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50690 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43360 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44618 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44634 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:27:12,598 - lava_callback - INFO - Received callback for node 69c64d62b98d4d3c02068bd8 INFO: 172.21.0.1:48802 - "POST /node/69c64d62b98d4d3c02068bd8 HTTP/1.0" 202 Accepted 2026-03-29 09:27:12,603 - lava_callback - INFO - Starting async job submit for node 69c64d62b98d4d3c02068bd8 2026-03-29 09:27:12,604 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:27:12,618 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64d62b98d4d3c02068bd8 HTTP/1.1" 200 None 2026-03-29 09:27:12,619 - lava_callback - DEBUG - Processing node 69c64d62b98d4d3c02068bd8: baseline-arm-baylibre 2026-03-29 09:27:12,619 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64d62b98d4d3c02068bd8 2026-03-29 09:27:12,620 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:27:12,638 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64d62b98d4d3c02068bd8 HTTP/1.1" 200 None 2026-03-29 09:27:12,639 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:27:12,652 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c0f72facdb2e26c83cca20 HTTP/1.1" 200 None 2026-03-29 09:27:12,654 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:27:12,674 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64d62b98d4d3c02068bd8 HTTP/1.1" 200 None 2026-03-29 09:27:12,675 - lava_callback - ERROR - Error processing callback for node 69c64d62b98d4d3c02068bd8: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:48808 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48818 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:27:17,793 - lava_callback - INFO - Received callback for node 69c8f03c6a4c026e16348afb INFO: 172.21.0.1:48826 - "POST /node/69c8f03c6a4c026e16348afb HTTP/1.0" 202 Accepted 2026-03-29 09:27:17,800 - lava_callback - INFO - Starting async job submit for node 69c8f03c6a4c026e16348afb 2026-03-29 09:27:17,829 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:27:17,843 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8f03c6a4c026e16348afb HTTP/1.1" 200 None 2026-03-29 09:27:17,844 - lava_callback - DEBUG - Processing node 69c8f03c6a4c026e16348afb: baseline-arm64 2026-03-29 09:27:18,023 - lava_callback - WARNING - Failed to upload log 2026-03-29 09:27:18,094 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:27:18,288 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:27:18,290 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8f03c6a4c026e16348afb/lava-logs.yaml 2026-03-29 09:27:18,293 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:27:18,438 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:27:18,440 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8f03c6a4c026e16348afb/lava_callback.json.gz 2026-03-29 09:27:18,470 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:27:18,490 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8f03c6a4c026e16348afb HTTP/1.1" 200 None 2026-03-29 09:27:18,491 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:27:18,505 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 09:27:18,506 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:27:18,527 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8f03c6a4c026e16348afb HTTP/1.1" 200 None 2026-03-29 09:27:18,607 - lava_callback - INFO - Completed processing callback for node 69c8f03c6a4c026e16348afb INFO: 172.21.0.1:49354 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56336 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:27:40,056 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:27:40,073 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:45950 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55412 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51602 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51614 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:28:12,726 - lava_callback - INFO - Received callback for node 69c64dabb98d4d3c02068bee INFO: 172.21.0.1:38186 - "POST /node/69c64dabb98d4d3c02068bee HTTP/1.0" 202 Accepted 2026-03-29 09:28:12,732 - lava_callback - INFO - Starting async job submit for node 69c64dabb98d4d3c02068bee 2026-03-29 09:28:12,733 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:28:12,756 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64dabb98d4d3c02068bee HTTP/1.1" 200 None 2026-03-29 09:28:12,756 - lava_callback - DEBUG - Processing node 69c64dabb98d4d3c02068bee: baseline-arm-baylibre 2026-03-29 09:28:12,757 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64dabb98d4d3c02068bee 2026-03-29 09:28:12,757 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:28:12,770 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64dabb98d4d3c02068bee HTTP/1.1" 200 None 2026-03-29 09:28:12,772 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:28:12,785 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c0f72facdb2e26c83cca20 HTTP/1.1" 200 None 2026-03-29 09:28:12,786 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:28:12,806 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64dabb98d4d3c02068bee HTTP/1.1" 200 None 2026-03-29 09:28:12,807 - lava_callback - ERROR - Error processing callback for node 69c64dabb98d4d3c02068bee: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:38192 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:38200 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33354 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58608 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57560 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:28:52,655 - lava_callback - INFO - Received callback for node 69c64dcfb98d4d3c02068bfb INFO: 172.21.0.1:44090 - "POST /node/69c64dcfb98d4d3c02068bfb HTTP/1.0" 202 Accepted 2026-03-29 09:28:52,661 - lava_callback - INFO - Starting async job submit for node 69c64dcfb98d4d3c02068bfb 2026-03-29 09:28:52,662 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:28:52,676 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64dcfb98d4d3c02068bfb HTTP/1.1" 200 None 2026-03-29 09:28:52,677 - lava_callback - DEBUG - Processing node 69c64dcfb98d4d3c02068bfb: baseline-arm-baylibre 2026-03-29 09:28:52,677 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64dcfb98d4d3c02068bfb 2026-03-29 09:28:52,678 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:28:52,695 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64dcfb98d4d3c02068bfb HTTP/1.1" 200 None 2026-03-29 09:28:52,696 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:28:52,711 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c0f72facdb2e26c83cca20 HTTP/1.1" 200 None 2026-03-29 09:28:52,713 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:28:52,734 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64dcfb98d4d3c02068bfb HTTP/1.1" 200 None 2026-03-29 09:28:52,740 - lava_callback - ERROR - Error processing callback for node 69c64dcfb98d4d3c02068bfb: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:44100 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50720 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50736 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43122 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:43132 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58768 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60830 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:29:45,852 - lava_callback - INFO - Received callback for node 69c64df7b98d4d3c02068c09 INFO: 172.21.0.1:41950 - "POST /node/69c64df7b98d4d3c02068c09 HTTP/1.0" 202 Accepted 2026-03-29 09:29:45,857 - lava_callback - INFO - Starting async job submit for node 69c64df7b98d4d3c02068c09 2026-03-29 09:29:45,858 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:29:45,872 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64df7b98d4d3c02068c09 HTTP/1.1" 200 None 2026-03-29 09:29:45,873 - lava_callback - DEBUG - Processing node 69c64df7b98d4d3c02068c09: baseline-arm-clabbe 2026-03-29 09:29:45,873 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64df7b98d4d3c02068c09 2026-03-29 09:29:45,874 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:29:45,887 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64df7b98d4d3c02068c09 HTTP/1.1" 200 None 2026-03-29 09:29:45,888 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:29:45,901 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69be544f448e074b3ffe162d HTTP/1.1" 200 None 2026-03-29 09:29:45,903 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:29:45,924 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64df7b98d4d3c02068c09 HTTP/1.1" 200 None 2026-03-29 09:29:45,937 - lava_callback - ERROR - Error processing callback for node 69c64df7b98d4d3c02068c09: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:41958 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42980 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45018 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45022 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50018 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:50022 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43730 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53770 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53156 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47060 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43562 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43566 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36644 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:36656 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48034 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:31:32,869 - lava_callback - INFO - Received callback for node 69c64e5fb98d4d3c02068c21 INFO: 172.21.0.1:54448 - "POST /node/69c64e5fb98d4d3c02068c21 HTTP/1.0" 202 Accepted 2026-03-29 09:31:32,874 - lava_callback - INFO - Starting async job submit for node 69c64e5fb98d4d3c02068c21 2026-03-29 09:31:32,875 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:31:32,891 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64e5fb98d4d3c02068c21 HTTP/1.1" 200 None 2026-03-29 09:31:32,892 - lava_callback - DEBUG - Processing node 69c64e5fb98d4d3c02068c21: baseline-arm-baylibre 2026-03-29 09:31:32,892 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64e5fb98d4d3c02068c21 2026-03-29 09:31:32,893 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:31:32,906 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64e5fb98d4d3c02068c21 HTTP/1.1" 200 None 2026-03-29 09:31:32,908 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:31:32,921 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c0f72facdb2e26c83cca20 HTTP/1.1" 200 None 2026-03-29 09:31:32,923 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:31:32,943 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64e5fb98d4d3c02068c21 HTTP/1.1" 200 None 2026-03-29 09:31:32,954 - lava_callback - ERROR - Error processing callback for node 69c64e5fb98d4d3c02068c21: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:54454 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58062 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:31:53,235 - lava_callback - INFO - Received callback for node 69c64e85b98d4d3c02068c33 INFO: 172.21.0.1:35624 - "POST /node/69c64e85b98d4d3c02068c33 HTTP/1.0" 202 Accepted 2026-03-29 09:31:53,241 - lava_callback - INFO - Starting async job submit for node 69c64e85b98d4d3c02068c33 2026-03-29 09:31:53,242 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:31:53,256 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64e85b98d4d3c02068c33 HTTP/1.1" 200 None 2026-03-29 09:31:53,257 - lava_callback - DEBUG - Processing node 69c64e85b98d4d3c02068c33: baseline-arm-baylibre 2026-03-29 09:31:53,257 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64e85b98d4d3c02068c33 2026-03-29 09:31:53,258 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:31:53,271 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64e85b98d4d3c02068c33 HTTP/1.1" 200 None 2026-03-29 09:31:53,272 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:31:53,286 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c0f72facdb2e26c83cca20 HTTP/1.1" 200 None 2026-03-29 09:31:53,288 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:31:53,307 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64e85b98d4d3c02068c33 HTTP/1.1" 200 None 2026-03-29 09:31:53,313 - lava_callback - ERROR - Error processing callback for node 69c64e85b98d4d3c02068c33: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:35632 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45230 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45234 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49646 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:49658 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:32:26,364 - lava_callback - INFO - Received callback for node 69c8ed706a4c026e16348a56 INFO: 172.21.0.1:58290 - "POST /node/69c8ed706a4c026e16348a56 HTTP/1.0" 202 Accepted 2026-03-29 09:32:26,371 - lava_callback - INFO - Starting async job submit for node 69c8ed706a4c026e16348a56 2026-03-29 09:32:26,427 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:32:26,442 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8ed706a4c026e16348a56 HTTP/1.1" 200 None 2026-03-29 09:32:26,442 - lava_callback - DEBUG - Processing node 69c8ed706a4c026e16348a56: baseline-arm INFO: 172.21.0.1:58298 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:32:27,107 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:32:27,255 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:32:27,257 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69c8ed706a4c026e16348a56/log.txt.gz 2026-03-29 09:32:27,644 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:32:27,882 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:32:27,884 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69c8ed706a4c026e16348a56/lava-logs.yaml 2026-03-29 09:32:27,899 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:32:28,074 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:32:28,076 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69c8ed706a4c026e16348a56/lava_callback.json.gz DEBUG: Setup failed for 69c8ed706a4c026e16348a56. Transitting job node result: incomplete -> fail 2026-03-29 09:32:28,186 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:32:28,201 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8ed706a4c026e16348a56 HTTP/1.1" 200 None 2026-03-29 09:32:28,202 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:32:28,216 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 09:32:28,218 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:32:28,257 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8ed706a4c026e16348a56 HTTP/1.1" 200 None 2026-03-29 09:32:28,382 - lava_callback - INFO - Completed processing callback for node 69c8ed706a4c026e16348a56 INFO: 172.21.0.1:56798 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:32:40,076 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:32:40,103 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-29 09:32:46,004 - lava_callback - INFO - Received callback for node 69c64eaab98d4d3c02068c43 INFO: 172.21.0.1:33610 - "POST /node/69c64eaab98d4d3c02068c43 HTTP/1.0" 202 Accepted 2026-03-29 09:32:46,010 - lava_callback - INFO - Starting async job submit for node 69c64eaab98d4d3c02068c43 2026-03-29 09:32:46,011 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:32:46,025 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64eaab98d4d3c02068c43 HTTP/1.1" 200 None 2026-03-29 09:32:46,026 - lava_callback - DEBUG - Processing node 69c64eaab98d4d3c02068c43: baseline-arm-clabbe 2026-03-29 09:32:46,026 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64eaab98d4d3c02068c43 2026-03-29 09:32:46,027 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:32:46,040 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64eaab98d4d3c02068c43 HTTP/1.1" 200 None 2026-03-29 09:32:46,041 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:32:46,054 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c0f72facdb2e26c83cca20 HTTP/1.1" 200 None 2026-03-29 09:32:46,056 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:32:46,089 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64eaab98d4d3c02068c43 HTTP/1.1" 200 None 2026-03-29 09:32:46,090 - lava_callback - ERROR - Error processing callback for node 69c64eaab98d4d3c02068c43: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:33614 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32792 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56118 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:33:05,971 - lava_callback - INFO - Received callback for node 69c64ecfb98d4d3c02068c53 INFO: 172.21.0.1:56128 - "POST /node/69c64ecfb98d4d3c02068c53 HTTP/1.0" 202 Accepted 2026-03-29 09:33:05,977 - lava_callback - INFO - Starting async job submit for node 69c64ecfb98d4d3c02068c53 2026-03-29 09:33:05,978 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:33:05,992 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64ecfb98d4d3c02068c53 HTTP/1.1" 200 None 2026-03-29 09:33:05,993 - lava_callback - DEBUG - Processing node 69c64ecfb98d4d3c02068c53: baseline-arm-clabbe 2026-03-29 09:33:05,993 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64ecfb98d4d3c02068c53 2026-03-29 09:33:05,994 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:33:06,010 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64ecfb98d4d3c02068c53 HTTP/1.1" 200 None 2026-03-29 09:33:06,012 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:33:06,026 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c0f72facdb2e26c83cca20 HTTP/1.1" 200 None 2026-03-29 09:33:06,027 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:33:06,053 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64ecfb98d4d3c02068c53 HTTP/1.1" 200 None 2026-03-29 09:33:06,054 - lava_callback - ERROR - Error processing callback for node 69c64ecfb98d4d3c02068c53: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:56134 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46888 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:46896 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36330 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42942 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:33:46,078 - lava_callback - INFO - Received callback for node 69c64ef4b98d4d3c02068c63 INFO: 172.21.0.1:43664 - "POST /node/69c64ef4b98d4d3c02068c63 HTTP/1.0" 202 Accepted 2026-03-29 09:33:46,083 - lava_callback - INFO - Starting async job submit for node 69c64ef4b98d4d3c02068c63 2026-03-29 09:33:46,084 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:33:46,106 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64ef4b98d4d3c02068c63 HTTP/1.1" 200 None 2026-03-29 09:33:46,106 - lava_callback - DEBUG - Processing node 69c64ef4b98d4d3c02068c63: baseline-arm-clabbe 2026-03-29 09:33:46,107 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64ef4b98d4d3c02068c63 2026-03-29 09:33:46,107 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:33:46,121 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64ef4b98d4d3c02068c63 HTTP/1.1" 200 None 2026-03-29 09:33:46,122 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:33:46,135 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c0f72facdb2e26c83cca20 HTTP/1.1" 200 None 2026-03-29 09:33:46,137 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:33:46,158 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64ef4b98d4d3c02068c63 HTTP/1.1" 200 None 2026-03-29 09:33:46,164 - lava_callback - ERROR - Error processing callback for node 69c64ef4b98d4d3c02068c63: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:43672 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56092 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34610 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34616 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44896 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:44902 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:34:26,034 - lava_callback - INFO - Received callback for node 69c64f19b98d4d3c02068c78 INFO: 172.21.0.1:44016 - "POST /node/69c64f19b98d4d3c02068c78 HTTP/1.0" 202 Accepted 2026-03-29 09:34:26,040 - lava_callback - INFO - Starting async job submit for node 69c64f19b98d4d3c02068c78 2026-03-29 09:34:26,041 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:34:26,055 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64f19b98d4d3c02068c78 HTTP/1.1" 200 None 2026-03-29 09:34:26,056 - lava_callback - DEBUG - Processing node 69c64f19b98d4d3c02068c78: baseline-arm-clabbe 2026-03-29 09:34:26,056 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64f19b98d4d3c02068c78 2026-03-29 09:34:26,056 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:34:26,070 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64f19b98d4d3c02068c78 HTTP/1.1" 200 None 2026-03-29 09:34:26,072 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:34:26,086 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c0f72facdb2e26c83cca20 HTTP/1.1" 200 None 2026-03-29 09:34:26,087 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:34:26,112 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64f19b98d4d3c02068c78 HTTP/1.1" 200 None 2026-03-29 09:34:26,113 - lava_callback - ERROR - Error processing callback for node 69c64f19b98d4d3c02068c78: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:44028 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37736 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32898 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33478 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53320 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:35:06,104 - lava_callback - INFO - Received callback for node 69c64f3eb98d4d3c02068c81 INFO: 172.21.0.1:53336 - "POST /node/69c64f3eb98d4d3c02068c81 HTTP/1.0" 202 Accepted 2026-03-29 09:35:06,109 - lava_callback - INFO - Starting async job submit for node 69c64f3eb98d4d3c02068c81 2026-03-29 09:35:06,110 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:35:06,146 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64f3eb98d4d3c02068c81 HTTP/1.1" 200 None 2026-03-29 09:35:06,147 - lava_callback - DEBUG - Processing node 69c64f3eb98d4d3c02068c81: baseline-arm-clabbe 2026-03-29 09:35:06,147 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64f3eb98d4d3c02068c81 2026-03-29 09:35:06,148 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:35:06,162 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64f3eb98d4d3c02068c81 HTTP/1.1" 200 None 2026-03-29 09:35:06,164 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:35:06,181 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c0f72facdb2e26c83cca20 HTTP/1.1" 200 None 2026-03-29 09:35:06,183 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:35:06,212 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64f3eb98d4d3c02068c81 HTTP/1.1" 200 None 2026-03-29 09:35:06,213 - lava_callback - ERROR - Error processing callback for node 69c64f3eb98d4d3c02068c81: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:53342 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:35:12,084 - lava_callback - INFO - Received callback for node 69c8edc36a4c026e16348a77 INFO: 172.21.0.1:58496 - "POST /node/69c8edc36a4c026e16348a77 HTTP/1.0" 202 Accepted 2026-03-29 09:35:12,133 - lava_callback - INFO - Starting async job submit for node 69c8edc36a4c026e16348a77 2026-03-29 09:35:12,157 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:35:12,177 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8edc36a4c026e16348a77 HTTP/1.1" 200 None 2026-03-29 09:35:12,177 - lava_callback - DEBUG - Processing node 69c8edc36a4c026e16348a77: baseline-arm-pengutronix 2026-03-29 09:35:12,282 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:35:12,477 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:35:12,479 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c8edc36a4c026e16348a77/log.txt.gz 2026-03-29 09:35:12,537 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:35:12,725 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:35:12,727 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c8edc36a4c026e16348a77/lava-logs.yaml 2026-03-29 09:35:12,730 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:35:12,876 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:35:12,878 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c8edc36a4c026e16348a77/lava_callback.json.gz DEBUG: Setup failed for 69c8edc36a4c026e16348a77. Transitting job node result: incomplete -> fail 2026-03-29 09:35:12,937 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:35:12,967 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8edc36a4c026e16348a77 HTTP/1.1" 200 None 2026-03-29 09:35:12,968 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:35:12,981 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 09:35:12,983 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:35:13,006 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8edc36a4c026e16348a77 HTTP/1.1" 200 None 2026-03-29 09:35:13,065 - lava_callback - INFO - Completed processing callback for node 69c8edc36a4c026e16348a77 INFO: 172.21.0.1:58498 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:58500 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36944 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36774 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:35:37,561 - lava_callback - INFO - Received callback for node 69c8e8276a4c026e1634871b INFO: 172.21.0.1:36776 - "POST /node/69c8e8276a4c026e1634871b HTTP/1.0" 202 Accepted 2026-03-29 09:35:37,568 - lava_callback - INFO - Starting async job submit for node 69c8e8276a4c026e1634871b 2026-03-29 09:35:37,593 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:35:37,608 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8276a4c026e1634871b HTTP/1.1" 200 None 2026-03-29 09:35:37,609 - lava_callback - DEBUG - Processing node 69c8e8276a4c026e1634871b: baseline-nfs-arm64 2026-03-29 09:35:38,152 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:35:38,322 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:35:38,324 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e8276a4c026e1634871b/log.txt.gz 2026-03-29 09:35:38,587 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:35:38,847 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:35:38,849 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e8276a4c026e1634871b/lava-logs.yaml 2026-03-29 09:35:38,861 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:35:39,019 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:35:39,021 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8e8276a4c026e1634871b/lava_callback.json.gz 2026-03-29 09:35:39,047 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:35:39,062 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e8276a4c026e1634871b HTTP/1.1" 200 None 2026-03-29 09:35:39,064 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:35:39,077 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 09:35:39,079 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:35:39,100 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e8276a4c026e1634871b HTTP/1.1" 200 None 2026-03-29 09:35:39,149 - lava_callback - INFO - Completed processing callback for node 69c8e8276a4c026e1634871b 2026-03-29 09:35:40,105 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:35:40,121 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-29 09:35:46,080 - lava_callback - INFO - Received callback for node 69c64f62b98d4d3c02068c90 INFO: 172.21.0.1:48490 - "POST /node/69c64f62b98d4d3c02068c90 HTTP/1.0" 202 Accepted 2026-03-29 09:35:46,085 - lava_callback - INFO - Starting async job submit for node 69c64f62b98d4d3c02068c90 2026-03-29 09:35:46,086 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:35:46,101 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64f62b98d4d3c02068c90 HTTP/1.1" 200 None 2026-03-29 09:35:46,102 - lava_callback - DEBUG - Processing node 69c64f62b98d4d3c02068c90: baseline-arm-clabbe 2026-03-29 09:35:46,102 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64f62b98d4d3c02068c90 2026-03-29 09:35:46,103 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:35:46,116 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64f62b98d4d3c02068c90 HTTP/1.1" 200 None 2026-03-29 09:35:46,117 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:35:46,130 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69be544f448e074b3ffe162d HTTP/1.1" 200 None 2026-03-29 09:35:46,131 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:35:46,157 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64f62b98d4d3c02068c90 HTTP/1.1" 200 None 2026-03-29 09:35:46,158 - lava_callback - ERROR - Error processing callback for node 69c64f62b98d4d3c02068c90: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:48506 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46402 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41384 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41386 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56074 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:56078 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47418 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35740 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38360 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47984 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50744 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50750 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35074 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:35084 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52330 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:37:33,194 - lava_callback - INFO - Received callback for node 69c64fd2b98d4d3c02068cc1 INFO: 172.21.0.1:40580 - "POST /node/69c64fd2b98d4d3c02068cc1 HTTP/1.0" 202 Accepted 2026-03-29 09:37:33,199 - lava_callback - INFO - Starting async job submit for node 69c64fd2b98d4d3c02068cc1 2026-03-29 09:37:33,200 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:37:33,243 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64fd2b98d4d3c02068cc1 HTTP/1.1" 200 None 2026-03-29 09:37:33,244 - lava_callback - DEBUG - Processing node 69c64fd2b98d4d3c02068cc1: baseline-arm64-baylibre 2026-03-29 09:37:33,244 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c64fd2b98d4d3c02068cc1 2026-03-29 09:37:33,245 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:37:33,258 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c64fd2b98d4d3c02068cc1 HTTP/1.1" 200 None 2026-03-29 09:37:33,259 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:37:33,273 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c0f73bacdb2e26c83cca2f HTTP/1.1" 200 None 2026-03-29 09:37:33,274 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:37:33,295 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c64fd2b98d4d3c02068cc1 HTTP/1.1" 200 None 2026-03-29 09:37:33,296 - lava_callback - ERROR - Error processing callback for node 69c64fd2b98d4d3c02068cc1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:40590 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53034 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54628 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47660 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47666 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37794 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:37808 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51812 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49750 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40782 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38698 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57904 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57918 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46586 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:46588 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34468 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:39:33,302 - lava_callback - INFO - Received callback for node 69c65042b98d4d3c02068ce1 INFO: 172.21.0.1:56120 - "POST /node/69c65042b98d4d3c02068ce1 HTTP/1.0" 202 Accepted 2026-03-29 09:39:33,308 - lava_callback - INFO - Starting async job submit for node 69c65042b98d4d3c02068ce1 2026-03-29 09:39:33,309 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:39:33,324 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c65042b98d4d3c02068ce1 HTTP/1.1" 200 None 2026-03-29 09:39:33,325 - lava_callback - DEBUG - Processing node 69c65042b98d4d3c02068ce1: baseline-arm64-baylibre 2026-03-29 09:39:33,325 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c65042b98d4d3c02068ce1 2026-03-29 09:39:33,326 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:39:33,339 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c65042b98d4d3c02068ce1 HTTP/1.1" 200 None 2026-03-29 09:39:33,340 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:39:33,353 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c0f73bacdb2e26c83cca2f HTTP/1.1" 200 None 2026-03-29 09:39:33,355 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:39:33,375 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c65042b98d4d3c02068ce1 HTTP/1.1" 200 None 2026-03-29 09:39:33,377 - lava_callback - ERROR - Error processing callback for node 69c65042b98d4d3c02068ce1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:56128 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58730 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34796 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50444 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50458 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54350 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:54352 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38052 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58838 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48204 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34080 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40982 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40990 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55576 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:55584 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41112 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35286 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48158 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56574 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43878 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43880 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32906 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:32912 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43970 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:42:33,442 - lava_callback - INFO - Received callback for node 69c65101b98d4d3c02068d0f INFO: 172.21.0.1:45102 - "POST /node/69c65101b98d4d3c02068d0f HTTP/1.0" 202 Accepted 2026-03-29 09:42:33,448 - lava_callback - INFO - Starting async job submit for node 69c65101b98d4d3c02068d0f 2026-03-29 09:42:33,449 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:42:33,463 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c65101b98d4d3c02068d0f HTTP/1.1" 200 None 2026-03-29 09:42:33,464 - lava_callback - DEBUG - Processing node 69c65101b98d4d3c02068d0f: baseline-arm64-baylibre 2026-03-29 09:42:33,464 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c65101b98d4d3c02068d0f 2026-03-29 09:42:33,465 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:42:33,478 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c65101b98d4d3c02068d0f HTTP/1.1" 200 None 2026-03-29 09:42:33,480 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:42:33,493 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c0f73bacdb2e26c83cca2f HTTP/1.1" 200 None 2026-03-29 09:42:33,495 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:42:33,516 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c65101b98d4d3c02068d0f HTTP/1.1" 200 None 2026-03-29 09:42:33,517 - lava_callback - ERROR - Error processing callback for node 69c65101b98d4d3c02068d0f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:45106 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43632 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50846 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51558 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51572 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59180 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:59184 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45400 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59524 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55898 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:43:53,571 - lava_callback - INFO - Received callback for node 69c6514fb98d4d3c02068d1a INFO: 172.21.0.1:39230 - "POST /node/69c6514fb98d4d3c02068d1a HTTP/1.0" 202 Accepted 2026-03-29 09:43:53,576 - lava_callback - INFO - Starting async job submit for node 69c6514fb98d4d3c02068d1a 2026-03-29 09:43:53,577 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:43:53,591 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6514fb98d4d3c02068d1a HTTP/1.1" 200 None 2026-03-29 09:43:53,592 - lava_callback - DEBUG - Processing node 69c6514fb98d4d3c02068d1a: baseline-arm64-baylibre 2026-03-29 09:43:53,592 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c6514fb98d4d3c02068d1a 2026-03-29 09:43:53,593 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:43:53,606 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6514fb98d4d3c02068d1a HTTP/1.1" 200 None 2026-03-29 09:43:53,607 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:43:53,620 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c0f73bacdb2e26c83cca2f HTTP/1.1" 200 None 2026-03-29 09:43:53,622 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:43:53,651 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c6514fb98d4d3c02068d1a HTTP/1.1" 200 None 2026-03-29 09:43:53,653 - lava_callback - ERROR - Error processing callback for node 69c6514fb98d4d3c02068d1a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:39244 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59482 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59498 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39990 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:39994 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45224 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54882 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45342 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53786 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58494 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58500 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48746 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48752 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56350 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52190 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34764 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36066 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32860 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32862 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50384 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:50398 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59960 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54758 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51220 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53050 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57258 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57264 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44112 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:44114 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51078 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33908 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40624 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:47:47,163 - lava_callback - INFO - Received callback for node 69c8f50b6a4c026e16348bb5 INFO: 172.21.0.1:40634 - "POST /node/69c8f50b6a4c026e16348bb5 HTTP/1.0" 202 Accepted 2026-03-29 09:47:47,169 - lava_callback - INFO - Starting async job submit for node 69c8f50b6a4c026e16348bb5 2026-03-29 09:47:47,198 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:47:47,213 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8f50b6a4c026e16348bb5 HTTP/1.1" 200 None 2026-03-29 09:47:47,214 - lava_callback - DEBUG - Processing node 69c8f50b6a4c026e16348bb5: baseline-arm64 2026-03-29 09:47:47,356 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:47:47,501 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:47:47,503 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8f50b6a4c026e16348bb5/log.txt.gz 2026-03-29 09:47:47,598 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:47:47,794 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:47:47,796 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8f50b6a4c026e16348bb5/lava-logs.yaml 2026-03-29 09:47:47,799 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:47:48,094 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:47:48,096 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8f50b6a4c026e16348bb5/lava_callback.json.gz 2026-03-29 09:47:48,126 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:47:48,140 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8f50b6a4c026e16348bb5 HTTP/1.1" 200 None 2026-03-29 09:47:48,141 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:47:48,154 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 09:47:48,156 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:47:48,176 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8f50b6a4c026e16348bb5 HTTP/1.1" 200 None 2026-03-29 09:47:48,254 - lava_callback - INFO - Completed processing callback for node 69c8f50b6a4c026e16348bb5 INFO: 172.21.0.1:55856 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49456 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49462 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:48:10,125 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:48:10,151 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:53112 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:53118 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51632 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46114 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54198 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43500 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44190 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44204 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59214 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:59222 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33380 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51596 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41676 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47692 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37190 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37198 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48670 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48676 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37248 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52586 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33954 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48894 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34574 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34586 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57760 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:57774 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39844 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:51:33,979 - lava_callback - INFO - Received callback for node 69c65315b98d4d3c02068d8a INFO: 172.21.0.1:53318 - "POST /node/69c65315b98d4d3c02068d8a HTTP/1.0" 202 Accepted 2026-03-29 09:51:33,984 - lava_callback - INFO - Starting async job submit for node 69c65315b98d4d3c02068d8a 2026-03-29 09:51:33,985 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:51:33,999 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c65315b98d4d3c02068d8a HTTP/1.1" 200 None 2026-03-29 09:51:34,001 - lava_callback - DEBUG - Processing node 69c65315b98d4d3c02068d8a: baseline-arm64-baylibre 2026-03-29 09:51:34,001 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c65315b98d4d3c02068d8a 2026-03-29 09:51:34,002 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:51:34,018 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c65315b98d4d3c02068d8a HTTP/1.1" 200 None 2026-03-29 09:51:34,020 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:51:34,033 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c0f73bacdb2e26c83cca2f HTTP/1.1" 200 None 2026-03-29 09:51:34,035 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:51:34,055 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c65315b98d4d3c02068d8a HTTP/1.1" 200 None 2026-03-29 09:51:34,061 - lava_callback - ERROR - Error processing callback for node 69c65315b98d4d3c02068d8a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:53326 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53390 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:51:48,510 - lava_callback - INFO - Received callback for node 69c8f5f26a4c026e16348bd3 INFO: 172.21.0.1:53398 - "POST /node/69c8f5f26a4c026e16348bd3 HTTP/1.0" 202 Accepted 2026-03-29 09:51:48,516 - lava_callback - INFO - Starting async job submit for node 69c8f5f26a4c026e16348bd3 2026-03-29 09:51:48,539 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:51:48,553 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8f5f26a4c026e16348bd3 HTTP/1.1" 200 None 2026-03-29 09:51:48,553 - lava_callback - DEBUG - Processing node 69c8f5f26a4c026e16348bd3: baseline-arm64 2026-03-29 09:51:48,873 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:51:49,014 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:51:49,016 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8f5f26a4c026e16348bd3/log.txt.gz 2026-03-29 09:51:49,210 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:51:49,444 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:51:49,446 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8f5f26a4c026e16348bd3/lava-logs.yaml 2026-03-29 09:51:49,453 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 09:51:49,625 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 09:51:49,627 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8f5f26a4c026e16348bd3/lava_callback.json.gz 2026-03-29 09:51:49,674 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:51:49,688 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8f5f26a4c026e16348bd3 HTTP/1.1" 200 None 2026-03-29 09:51:49,690 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:51:49,703 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 09:51:49,705 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:51:49,729 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8f5f26a4c026e16348bd3 HTTP/1.1" 200 None 2026-03-29 09:51:49,789 - lava_callback - INFO - Completed processing callback for node 69c8f5f26a4c026e16348bd3 INFO: 172.21.0.1:42146 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44988 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44998 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:52:10,153 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:52:10,170 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:52984 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:52990 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49072 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53286 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48524 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50776 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40982 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40998 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50270 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:50286 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57676 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43704 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42490 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32792 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54736 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54740 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60968 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:60984 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37194 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34818 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54882 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33828 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55598 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55608 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49320 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:49328 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48508 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:55:26,953 - lava_callback - INFO - Received callback for node 69c653ffb98d4d3c02068dc1 INFO: 172.21.0.1:48516 - "POST /node/69c653ffb98d4d3c02068dc1 HTTP/1.0" 202 Accepted 2026-03-29 09:55:26,959 - lava_callback - INFO - Starting async job submit for node 69c653ffb98d4d3c02068dc1 2026-03-29 09:55:26,960 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:55:26,974 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c653ffb98d4d3c02068dc1 HTTP/1.1" 200 None 2026-03-29 09:55:26,975 - lava_callback - DEBUG - Processing node 69c653ffb98d4d3c02068dc1: baseline-arm64-clabbe 2026-03-29 09:55:26,975 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c653ffb98d4d3c02068dc1 2026-03-29 09:55:26,976 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:55:26,989 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c653ffb98d4d3c02068dc1 HTTP/1.1" 200 None 2026-03-29 09:55:26,990 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:55:27,003 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c0f73bacdb2e26c83cca2f HTTP/1.1" 200 None 2026-03-29 09:55:27,005 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:55:27,030 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c653ffb98d4d3c02068dc1 HTTP/1.1" 200 None 2026-03-29 09:55:27,031 - lava_callback - ERROR - Error processing callback for node 69c653ffb98d4d3c02068dc1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:60708 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54964 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50558 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50288 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50304 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54476 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:54478 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35874 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57532 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36796 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:56:47,100 - lava_callback - INFO - Received callback for node 69c6544cb98d4d3c02068dd0 INFO: 172.21.0.1:36800 - "POST /node/69c6544cb98d4d3c02068dd0 HTTP/1.0" 202 Accepted 2026-03-29 09:56:47,106 - lava_callback - INFO - Starting async job submit for node 69c6544cb98d4d3c02068dd0 2026-03-29 09:56:47,107 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:56:47,121 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6544cb98d4d3c02068dd0 HTTP/1.1" 200 None 2026-03-29 09:56:47,122 - lava_callback - DEBUG - Processing node 69c6544cb98d4d3c02068dd0: baseline-arm64-clabbe 2026-03-29 09:56:47,122 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c6544cb98d4d3c02068dd0 2026-03-29 09:56:47,123 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:56:47,136 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6544cb98d4d3c02068dd0 HTTP/1.1" 200 None 2026-03-29 09:56:47,137 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:56:47,154 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c0f73bacdb2e26c83cca2f HTTP/1.1" 200 None 2026-03-29 09:56:47,155 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:56:47,178 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c6544cb98d4d3c02068dd0 HTTP/1.1" 200 None 2026-03-29 09:56:47,184 - lava_callback - ERROR - Error processing callback for node 69c6544cb98d4d3c02068dd0: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:45154 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35598 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35614 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46588 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:46604 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44502 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41558 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39450 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36130 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59514 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59522 - "GET / HTTP/1.0" 200 OK 2026-03-29 09:58:07,097 - lava_callback - INFO - Received callback for node 69c6549ab98d4d3c02068de4 INFO: 172.21.0.1:59532 - "POST /node/69c6549ab98d4d3c02068de4 HTTP/1.0" 202 Accepted 2026-03-29 09:58:07,103 - lava_callback - INFO - Starting async job submit for node 69c6549ab98d4d3c02068de4 2026-03-29 09:58:07,104 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:58:07,119 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6549ab98d4d3c02068de4 HTTP/1.1" 200 None 2026-03-29 09:58:07,120 - lava_callback - DEBUG - Processing node 69c6549ab98d4d3c02068de4: baseline-arm64-clabbe 2026-03-29 09:58:07,120 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c6549ab98d4d3c02068de4 2026-03-29 09:58:07,121 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:58:07,135 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6549ab98d4d3c02068de4 HTTP/1.1" 200 None 2026-03-29 09:58:07,136 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:58:07,150 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c0f73bacdb2e26c83cca2f HTTP/1.1" 200 None 2026-03-29 09:58:07,152 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 09:58:07,190 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c6549ab98d4d3c02068de4 HTTP/1.1" 200 None 2026-03-29 09:58:07,191 - lava_callback - ERROR - Error processing callback for node 69c6549ab98d4d3c02068de4: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:57998 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:58008 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33734 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48340 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51794 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49956 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37278 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37280 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50440 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:50448 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60036 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55998 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55808 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59016 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56580 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56596 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54882 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:54890 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41406 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:00:27,338 - lava_callback - INFO - Received callback for node 69c65537b98d4d3c02068dfd INFO: 172.21.0.1:41412 - "POST /node/69c65537b98d4d3c02068dfd HTTP/1.0" 202 Accepted 2026-03-29 10:00:27,343 - lava_callback - INFO - Starting async job submit for node 69c65537b98d4d3c02068dfd 2026-03-29 10:00:27,344 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:00:27,359 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c65537b98d4d3c02068dfd HTTP/1.1" 200 None 2026-03-29 10:00:27,359 - lava_callback - DEBUG - Processing node 69c65537b98d4d3c02068dfd: baseline-arm64-clabbe 2026-03-29 10:00:27,360 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c65537b98d4d3c02068dfd 2026-03-29 10:00:27,360 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:00:27,374 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c65537b98d4d3c02068dfd HTTP/1.1" 200 None 2026-03-29 10:00:27,375 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:00:27,413 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c0f73bacdb2e26c83cca2f HTTP/1.1" 200 None 2026-03-29 10:00:27,415 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:00:27,440 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c65537b98d4d3c02068dfd HTTP/1.1" 200 None 2026-03-29 10:00:27,441 - lava_callback - ERROR - Error processing callback for node 69c65537b98d4d3c02068dfd: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:59422 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59188 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39860 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52464 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52480 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58972 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:58980 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52488 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34820 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33214 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53810 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33152 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33158 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44630 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:44636 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57924 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51574 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:02:37,719 - lava_callback - INFO - Received callback for node 69c8f8ae6a4c026e16348c30 INFO: 172.21.0.1:51588 - "POST /node/69c8f8ae6a4c026e16348c30 HTTP/1.0" 202 Accepted 2026-03-29 10:02:37,725 - lava_callback - INFO - Starting async job submit for node 69c8f8ae6a4c026e16348c30 2026-03-29 10:02:37,727 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:02:37,741 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8f8ae6a4c026e16348c30 HTTP/1.1" 200 None 2026-03-29 10:02:37,742 - lava_callback - DEBUG - Processing node 69c8f8ae6a4c026e16348c30: baseline-nfs-arm64 2026-03-29 10:02:37,744 - lava_callback - WARNING - Failed to upload log 2026-03-29 10:02:37,745 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:02:37,926 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:02:37,928 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8f8ae6a4c026e16348c30/lava-logs.yaml 2026-03-29 10:02:37,929 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:02:38,063 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:02:38,065 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8f8ae6a4c026e16348c30/lava_callback.json.gz 2026-03-29 10:02:38,067 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:02:38,081 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8f8ae6a4c026e16348c30 HTTP/1.1" 200 None 2026-03-29 10:02:38,083 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:02:38,097 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 10:02:38,098 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:02:38,128 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8f8ae6a4c026e16348c30 HTTP/1.1" 200 None 2026-03-29 10:02:38,132 - lava_callback - INFO - Completed processing callback for node 69c8f8ae6a4c026e16348c30 2026-03-29 10:02:40,174 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:02:40,189 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:39780 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55856 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46064 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46076 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:03:07,308 - lava_callback - INFO - Received callback for node 69c655d2b98d4d3c02068e17 INFO: 172.21.0.1:46088 - "POST /node/69c655d2b98d4d3c02068e17 HTTP/1.0" 202 Accepted 2026-03-29 10:03:07,314 - lava_callback - INFO - Starting async job submit for node 69c655d2b98d4d3c02068e17 2026-03-29 10:03:07,315 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:03:07,330 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c655d2b98d4d3c02068e17 HTTP/1.1" 200 None 2026-03-29 10:03:07,331 - lava_callback - DEBUG - Processing node 69c655d2b98d4d3c02068e17: baseline-arm64-clabbe 2026-03-29 10:03:07,331 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c655d2b98d4d3c02068e17 2026-03-29 10:03:07,332 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:03:07,353 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c655d2b98d4d3c02068e17 HTTP/1.1" 200 None 2026-03-29 10:03:07,354 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:03:07,368 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c0f73bacdb2e26c83cca2f HTTP/1.1" 200 None 2026-03-29 10:03:07,370 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:03:07,396 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c655d2b98d4d3c02068e17 HTTP/1.1" 200 None 2026-03-29 10:03:07,397 - lava_callback - ERROR - Error processing callback for node 69c655d2b98d4d3c02068e17: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:43568 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:43578 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43038 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41612 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54436 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35910 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49294 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49304 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51648 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:51662 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53480 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:04:27,358 - lava_callback - INFO - Received callback for node 69c6561fb98d4d3c02068e28 INFO: 172.21.0.1:53492 - "POST /node/69c6561fb98d4d3c02068e28 HTTP/1.0" 202 Accepted 2026-03-29 10:04:27,364 - lava_callback - INFO - Starting async job submit for node 69c6561fb98d4d3c02068e28 2026-03-29 10:04:27,365 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:04:27,378 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6561fb98d4d3c02068e28 HTTP/1.1" 200 None 2026-03-29 10:04:27,379 - lava_callback - DEBUG - Processing node 69c6561fb98d4d3c02068e28: baseline-arm64-clabbe 2026-03-29 10:04:27,379 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c6561fb98d4d3c02068e28 2026-03-29 10:04:27,380 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:04:27,393 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6561fb98d4d3c02068e28 HTTP/1.1" 200 None 2026-03-29 10:04:27,394 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:04:27,407 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c0f73bacdb2e26c83cca2f HTTP/1.1" 200 None 2026-03-29 10:04:27,408 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:04:27,439 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c6561fb98d4d3c02068e28 HTTP/1.1" 200 None 2026-03-29 10:04:27,440 - lava_callback - ERROR - Error processing callback for node 69c6561fb98d4d3c02068e28: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:54754 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43846 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43870 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32988 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33000 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49234 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:49246 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33742 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:05:30,767 - lava_callback - INFO - Received callback for node 69c8f4c56a4c026e16348bae INFO: 172.21.0.1:51380 - "POST /node/69c8f4c56a4c026e16348bae HTTP/1.0" 202 Accepted 2026-03-29 10:05:30,775 - lava_callback - INFO - Starting async job submit for node 69c8f4c56a4c026e16348bae 2026-03-29 10:05:30,818 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:05:30,836 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8f4c56a4c026e16348bae HTTP/1.1" 200 None 2026-03-29 10:05:30,837 - lava_callback - DEBUG - Processing node 69c8f4c56a4c026e16348bae: baseline-nfs-arm64 2026-03-29 10:05:31,892 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:05:32,109 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:05:32,110 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8f4c56a4c026e16348bae/log.txt.gz 2026-03-29 10:05:32,706 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:05:32,944 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:05:32,946 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8f4c56a4c026e16348bae/lava-logs.yaml 2026-03-29 10:05:32,967 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:05:33,217 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:05:33,219 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8f4c56a4c026e16348bae/lava_callback.json.gz DEBUG: Setup failed for 69c8f4c56a4c026e16348bae. Transitting job node result: incomplete -> fail 2026-03-29 10:05:33,317 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:05:33,331 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8f4c56a4c026e16348bae HTTP/1.1" 200 None 2026-03-29 10:05:33,333 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:05:33,346 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 10:05:33,347 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:05:33,380 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8f4c56a4c026e16348bae HTTP/1.1" 200 None 2026-03-29 10:05:33,470 - lava_callback - INFO - Completed processing callback for node 69c8f4c56a4c026e16348bae INFO: 172.21.0.1:51382 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:05:40,191 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:05:40,207 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:37852 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51690 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38262 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38278 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60848 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:60860 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49670 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59058 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43054 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37072 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39038 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39046 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46470 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:46482 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38786 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53836 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48950 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44458 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42352 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42354 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56416 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:56426 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37604 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48078 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34628 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58428 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35804 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35810 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52140 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:52154 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51912 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42490 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:09:40,849 - lava_callback - INFO - Received callback for node 69c8fa256a4c026e16348c70 INFO: 172.21.0.1:49482 - "POST /node/69c8fa256a4c026e16348c70 HTTP/1.0" 202 Accepted 2026-03-29 10:09:40,855 - lava_callback - INFO - Starting async job submit for node 69c8fa256a4c026e16348c70 2026-03-29 10:09:40,885 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:09:40,900 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8fa256a4c026e16348c70 HTTP/1.1" 200 None 2026-03-29 10:09:40,901 - lava_callback - DEBUG - Processing node 69c8fa256a4c026e16348c70: baseline-arm64 2026-03-29 10:09:41,038 - lava_callback - WARNING - Failed to upload log 2026-03-29 10:09:41,108 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:09:41,331 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:09:41,333 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8fa256a4c026e16348c70/lava-logs.yaml 2026-03-29 10:09:41,336 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:09:41,464 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:09:41,466 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8fa256a4c026e16348c70/lava_callback.json.gz 2026-03-29 10:09:41,499 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:09:41,517 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8fa256a4c026e16348c70 HTTP/1.1" 200 None 2026-03-29 10:09:41,518 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:09:41,534 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 10:09:41,536 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:09:41,567 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8fa256a4c026e16348c70 HTTP/1.1" 200 None 2026-03-29 10:09:41,643 - lava_callback - INFO - Completed processing callback for node 69c8fa256a4c026e16348c70 INFO: 172.21.0.1:49498 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35852 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47260 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47262 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:10:10,209 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:10:10,234 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:46850 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:46866 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52690 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47732 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48922 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52944 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58278 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58286 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37720 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:37726 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56262 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50540 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58974 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53332 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38344 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38346 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54236 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:54244 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58464 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51314 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55698 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42080 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47640 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47648 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48516 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48532 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35078 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41664 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:13:45,542 - lava_callback - INFO - Received callback for node 69c8f9916a4c026e16348c57 INFO: 172.21.0.1:52022 - "POST /node/69c8f9916a4c026e16348c57 HTTP/1.0" 202 Accepted 2026-03-29 10:13:45,549 - lava_callback - INFO - Starting async job submit for node 69c8f9916a4c026e16348c57 2026-03-29 10:13:45,585 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:13:45,602 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8f9916a4c026e16348c57 HTTP/1.1" 200 None 2026-03-29 10:13:45,603 - lava_callback - DEBUG - Processing node 69c8f9916a4c026e16348c57: baseline-arm64 2026-03-29 10:13:45,995 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:13:46,145 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:13:46,146 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8f9916a4c026e16348c57/log.txt.gz 2026-03-29 10:13:46,434 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:52036 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:13:46,657 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:13:46,659 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8f9916a4c026e16348c57/lava-logs.yaml 2026-03-29 10:13:46,668 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:13:46,825 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:13:46,827 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8f9916a4c026e16348c57/lava_callback.json.gz DEBUG: Setup failed for 69c8f9916a4c026e16348c57. Transitting job node result: incomplete -> fail 2026-03-29 10:13:46,901 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:13:46,925 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8f9916a4c026e16348c57 HTTP/1.1" 200 None 2026-03-29 10:13:46,926 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:13:46,939 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 10:13:46,941 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:13:46,966 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8f9916a4c026e16348c57 HTTP/1.1" 200 None 2026-03-29 10:13:47,056 - lava_callback - INFO - Completed processing callback for node 69c8f9916a4c026e16348c57 INFO: 172.21.0.1:54828 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60708 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60722 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:14:10,237 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:14:10,253 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:48428 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48440 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52844 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:14:26,956 - lava_callback - INFO - Received callback for node 69c8f8f76a4c026e16348c3f INFO: 172.21.0.1:52850 - "POST /node/69c8f8f76a4c026e16348c3f HTTP/1.0" 202 Accepted 2026-03-29 10:14:26,967 - lava_callback - INFO - Starting async job submit for node 69c8f8f76a4c026e16348c3f 2026-03-29 10:14:26,992 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:14:27,016 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8f8f76a4c026e16348c3f HTTP/1.1" 200 None 2026-03-29 10:14:27,017 - lava_callback - DEBUG - Processing node 69c8f8f76a4c026e16348c3f: baseline-arm64 2026-03-29 10:14:27,536 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:14:27,645 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:14:27,646 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8f8f76a4c026e16348c3f/log.txt.gz 2026-03-29 10:14:27,896 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:14:28,278 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:14:28,281 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8f8f76a4c026e16348c3f/lava-logs.yaml 2026-03-29 10:14:28,292 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:14:28,482 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:14:28,484 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8f8f76a4c026e16348c3f/lava_callback.json.gz 2026-03-29 10:14:28,509 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:14:28,523 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8f8f76a4c026e16348c3f HTTP/1.1" 200 None 2026-03-29 10:14:28,525 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:14:28,538 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 10:14:28,540 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:14:28,561 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8f8f76a4c026e16348c3f HTTP/1.1" 200 None 2026-03-29 10:14:28,620 - lava_callback - INFO - Completed processing callback for node 69c8f8f76a4c026e16348c3f INFO: 172.21.0.1:37292 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:14:40,255 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:14:40,271 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:35768 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49996 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52422 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52428 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33612 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:33620 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53656 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49548 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50876 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52614 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:16:04,155 - lava_callback - INFO - Received callback for node 69c8f7bd6a4c026e16348c12 INFO: 172.21.0.1:37316 - "POST /node/69c8f7bd6a4c026e16348c12 HTTP/1.0" 202 Accepted 2026-03-29 10:16:04,162 - lava_callback - INFO - Starting async job submit for node 69c8f7bd6a4c026e16348c12 2026-03-29 10:16:04,221 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:16:04,235 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8f7bd6a4c026e16348c12 HTTP/1.1" 200 None 2026-03-29 10:16:04,236 - lava_callback - DEBUG - Processing node 69c8f7bd6a4c026e16348c12: ltp-smoke 2026-03-29 10:16:04,757 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:16:04,969 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:16:04,971 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-smoke-69c8f7bd6a4c026e16348c12/log.txt.gz INFO: 172.21.0.1:37328 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:16:05,249 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:16:05,489 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:16:05,491 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-smoke-69c8f7bd6a4c026e16348c12/lava-logs.yaml 2026-03-29 10:16:05,502 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:16:05,684 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:16:05,686 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-smoke-69c8f7bd6a4c026e16348c12/lava_callback.json.gz 2026-03-29 10:16:05,806 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:16:05,821 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8f7bd6a4c026e16348c12 HTTP/1.1" 200 None 2026-03-29 10:16:05,822 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:16:05,835 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff06a4c026e16347d3c HTTP/1.1" 200 None 2026-03-29 10:16:05,837 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:16:05,871 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8f7bd6a4c026e16348c12 HTTP/1.1" 200 None 2026-03-29 10:16:05,990 - lava_callback - INFO - Completed processing callback for node 69c8f7bd6a4c026e16348c12 INFO: 172.21.0.1:37344 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:16:10,274 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:16:10,290 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:48624 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48628 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59868 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54086 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45718 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36014 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45890 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45900 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48340 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48346 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38810 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54616 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52032 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37150 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36194 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36206 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34234 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:34240 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52658 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52252 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42680 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38374 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49200 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49214 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48434 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48446 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59462 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36690 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39928 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54410 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39812 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39828 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42988 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:43002 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41522 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51230 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46738 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40286 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58736 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58746 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40070 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:40078 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45510 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47160 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53750 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53070 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36356 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36370 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47948 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47950 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36216 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34622 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50292 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54062 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38666 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38670 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60358 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:60360 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41164 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41888 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47824 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37184 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51188 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51194 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:24:07,910 - lava_callback - INFO - Received callback for node 69c8fd8a6a4c026e16348d17 INFO: 172.21.0.1:51204 - "POST /node/69c8fd8a6a4c026e16348d17 HTTP/1.0" 202 Accepted 2026-03-29 10:24:07,917 - lava_callback - INFO - Starting async job submit for node 69c8fd8a6a4c026e16348d17 2026-03-29 10:24:07,948 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:24:07,981 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8fd8a6a4c026e16348d17 HTTP/1.1" 200 None 2026-03-29 10:24:07,982 - lava_callback - DEBUG - Processing node 69c8fd8a6a4c026e16348d17: baseline-arm64 2026-03-29 10:24:08,125 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:24:08,288 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:24:08,290 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8fd8a6a4c026e16348d17/log.txt.gz 2026-03-29 10:24:08,364 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:24:08,547 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:24:08,549 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8fd8a6a4c026e16348d17/lava-logs.yaml 2026-03-29 10:24:08,553 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:24:08,693 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:24:08,695 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8fd8a6a4c026e16348d17/lava_callback.json.gz 2026-03-29 10:24:08,730 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:24:08,751 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8fd8a6a4c026e16348d17 HTTP/1.1" 200 None 2026-03-29 10:24:08,752 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:24:08,766 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 10:24:08,767 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:24:08,791 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8fd8a6a4c026e16348d17 HTTP/1.1" 200 None 2026-03-29 10:24:08,870 - lava_callback - INFO - Completed processing callback for node 69c8fd8a6a4c026e16348d17 2026-03-29 10:24:10,293 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:24:10,308 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:52494 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:52506 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57180 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33600 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59282 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37070 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37520 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37522 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54946 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:54948 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42312 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58810 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40660 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41320 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54466 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54482 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39496 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:39512 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50324 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52744 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:26:46,291 - lava_callback - INFO - Received callback for node 69c8fe226a4c026e16348d2a INFO: 172.21.0.1:53800 - "POST /node/69c8fe226a4c026e16348d2a HTTP/1.0" 202 Accepted 2026-03-29 10:26:46,297 - lava_callback - INFO - Starting async job submit for node 69c8fe226a4c026e16348d2a 2026-03-29 10:26:46,319 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:26:46,334 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8fe226a4c026e16348d2a HTTP/1.1" 200 None 2026-03-29 10:26:46,335 - lava_callback - DEBUG - Processing node 69c8fe226a4c026e16348d2a: baseline-arm64 INFO: 172.21.0.1:53814 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:26:46,683 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:26:46,835 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:26:46,837 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8fe226a4c026e16348d2a/log.txt.gz 2026-03-29 10:26:47,032 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:26:47,248 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:26:47,251 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8fe226a4c026e16348d2a/lava-logs.yaml 2026-03-29 10:26:47,258 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:26:47,415 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:26:47,417 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c8fe226a4c026e16348d2a/lava_callback.json.gz 2026-03-29 10:26:47,461 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:26:47,476 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8fe226a4c026e16348d2a HTTP/1.1" 200 None 2026-03-29 10:26:47,478 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:26:47,492 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 10:26:47,493 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:26:47,517 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8fe226a4c026e16348d2a HTTP/1.1" 200 None 2026-03-29 10:26:47,575 - lava_callback - INFO - Completed processing callback for node 69c8fe226a4c026e16348d2a INFO: 172.21.0.1:60008 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41138 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41152 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:27:10,311 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:27:10,326 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:36004 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:36018 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46872 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54598 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:27:39,189 - lava_callback - INFO - Received callback for node 69c8fc156a4c026e16348cef INFO: 172.21.0.1:54614 - "POST /node/69c8fc156a4c026e16348cef HTTP/1.0" 202 Accepted 2026-03-29 10:27:39,195 - lava_callback - INFO - Starting async job submit for node 69c8fc156a4c026e16348cef 2026-03-29 10:27:39,221 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:27:39,236 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8fc156a4c026e16348cef HTTP/1.1" 200 None 2026-03-29 10:27:39,237 - lava_callback - DEBUG - Processing node 69c8fc156a4c026e16348cef: baseline-nfs-arm64 2026-03-29 10:27:39,766 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:27:39,893 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:27:39,894 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8fc156a4c026e16348cef/log.txt.gz 2026-03-29 10:27:40,147 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:27:40,353 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:27:40,355 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8fc156a4c026e16348cef/lava-logs.yaml 2026-03-29 10:27:40,367 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:27:40,534 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:27:40,536 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c8fc156a4c026e16348cef/lava_callback.json.gz 2026-03-29 10:27:40,563 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:27:40,579 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8fc156a4c026e16348cef HTTP/1.1" 200 None 2026-03-29 10:27:40,580 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:27:40,594 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 10:27:40,595 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:27:40,615 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8fc156a4c026e16348cef HTTP/1.1" 200 None 2026-03-29 10:27:40,674 - lava_callback - INFO - Completed processing callback for node 69c8fc156a4c026e16348cef INFO: 172.21.0.1:60630 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47598 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55716 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55728 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:28:10,328 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:28:10,344 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:47826 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47830 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35240 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34162 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51410 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53540 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43882 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43892 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:29:07,360 - lava_callback - INFO - Received callback for node 69c8fb086a4c026e16348c9e INFO: 172.21.0.1:43908 - "POST /node/69c8fb086a4c026e16348c9e HTTP/1.0" 202 Accepted 2026-03-29 10:29:07,367 - lava_callback - INFO - Starting async job submit for node 69c8fb086a4c026e16348c9e 2026-03-29 10:29:07,420 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:29:07,434 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8fb086a4c026e16348c9e HTTP/1.1" 200 None 2026-03-29 10:29:07,435 - lava_callback - DEBUG - Processing node 69c8fb086a4c026e16348c9e: baseline-arm 2026-03-29 10:29:08,071 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:29:08,226 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:29:08,227 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69c8fb086a4c026e16348c9e/log.txt.gz 2026-03-29 10:29:08,578 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:29:08,849 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:29:08,851 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69c8fb086a4c026e16348c9e/lava-logs.yaml 2026-03-29 10:29:08,866 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:29:09,050 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:29:09,052 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69c8fb086a4c026e16348c9e/lava_callback.json.gz DEBUG: Setup failed for 69c8fb086a4c026e16348c9e. Transitting job node result: incomplete -> fail 2026-03-29 10:29:09,161 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:29:09,176 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8fb086a4c026e16348c9e HTTP/1.1" 200 None 2026-03-29 10:29:09,177 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:29:09,191 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 10:29:09,193 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:29:09,234 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8fb086a4c026e16348c9e HTTP/1.1" 200 None 2026-03-29 10:29:09,342 - lava_callback - INFO - Completed processing callback for node 69c8fb086a4c026e16348c9e 2026-03-29 10:29:10,345 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:29:10,361 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:51204 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:51214 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48090 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48284 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36140 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48538 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48920 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48936 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47002 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47008 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49858 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49028 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51072 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59266 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55178 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55184 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36794 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:36800 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33512 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46806 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45212 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46302 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43584 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43586 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43880 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:43884 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60276 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55732 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42498 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60388 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33328 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33334 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53742 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:53754 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51054 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60598 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33230 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43816 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49640 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49644 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36398 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:36408 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44192 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55678 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58712 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43362 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51780 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51792 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41680 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:41686 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50066 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47764 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34498 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:35:54,787 - lava_callback - INFO - Received callback for node 69c8fbf26a4c026e16348ce6 INFO: 172.21.0.1:52552 - "POST /node/69c8fbf26a4c026e16348ce6 HTTP/1.0" 202 Accepted 2026-03-29 10:35:54,833 - lava_callback - INFO - Starting async job submit for node 69c8fbf26a4c026e16348ce6 2026-03-29 10:35:54,857 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:35:54,871 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8fbf26a4c026e16348ce6 HTTP/1.1" 200 None 2026-03-29 10:35:54,872 - lava_callback - DEBUG - Processing node 69c8fbf26a4c026e16348ce6: baseline-arm-pengutronix 2026-03-29 10:35:54,978 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:35:55,142 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:35:55,143 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c8fbf26a4c026e16348ce6/log.txt.gz 2026-03-29 10:35:55,201 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:35:55,400 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:35:55,401 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c8fbf26a4c026e16348ce6/lava-logs.yaml 2026-03-29 10:35:55,405 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:35:55,524 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:35:55,525 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c8fbf26a4c026e16348ce6/lava_callback.json.gz DEBUG: Setup failed for 69c8fbf26a4c026e16348ce6. Transitting job node result: incomplete -> fail 2026-03-29 10:35:55,584 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:35:55,601 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8fbf26a4c026e16348ce6 HTTP/1.1" 200 None 2026-03-29 10:35:55,603 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:35:55,637 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 10:35:55,639 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:35:55,665 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8fbf26a4c026e16348ce6 HTTP/1.1" 200 None 2026-03-29 10:35:55,732 - lava_callback - INFO - Completed processing callback for node 69c8fbf26a4c026e16348ce6 2026-03-29 10:35:56,418 - lava_callback - INFO - Received callback for node 69c65d82b98d4d3c02068fa3 INFO: 172.21.0.1:52566 - "POST /node/69c65d82b98d4d3c02068fa3 HTTP/1.0" 202 Accepted 2026-03-29 10:35:56,426 - lava_callback - INFO - Starting async job submit for node 69c65d82b98d4d3c02068fa3 2026-03-29 10:35:56,427 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:35:56,444 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c65d82b98d4d3c02068fa3 HTTP/1.1" 200 None 2026-03-29 10:35:56,445 - lava_callback - DEBUG - Processing node 69c65d82b98d4d3c02068fa3: baseline-arm64-baylibre 2026-03-29 10:35:56,446 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c65d82b98d4d3c02068fa3 2026-03-29 10:35:56,446 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:35:56,460 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c65d82b98d4d3c02068fa3 HTTP/1.1" 200 None 2026-03-29 10:35:56,461 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:35:56,474 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69be5461448e074b3ffe163c HTTP/1.1" 200 None 2026-03-29 10:35:56,475 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:35:56,495 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c65d82b98d4d3c02068fa3 HTTP/1.1" 200 None 2026-03-29 10:35:56,497 - lava_callback - ERROR - Error processing callback for node 69c65d82b98d4d3c02068fa3: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:52576 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50130 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50132 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:36:10,365 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:36:10,380 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:54170 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:54186 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58728 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57142 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35642 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44318 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58952 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58966 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55032 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-29 10:37:16,462 - lava_callback - INFO - Received callback for node 69c65dd0b98d4d3c02068faa INFO: 172.21.0.1:55034 - "POST /node/69c65dd0b98d4d3c02068faa HTTP/1.0" 202 Accepted 2026-03-29 10:37:16,467 - lava_callback - INFO - Starting async job submit for node 69c65dd0b98d4d3c02068faa 2026-03-29 10:37:16,468 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:37:16,482 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c65dd0b98d4d3c02068faa HTTP/1.1" 200 None 2026-03-29 10:37:16,483 - lava_callback - DEBUG - Processing node 69c65dd0b98d4d3c02068faa: baseline-arm64-baylibre 2026-03-29 10:37:16,484 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c65dd0b98d4d3c02068faa 2026-03-29 10:37:16,484 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:37:16,501 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c65dd0b98d4d3c02068faa HTTP/1.1" 200 None 2026-03-29 10:37:16,502 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 INFO: 172.21.0.1:55036 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:37:16,517 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69be5461448e074b3ffe163c HTTP/1.1" 200 None 2026-03-29 10:37:16,519 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:37:16,540 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c65dd0b98d4d3c02068faa HTTP/1.1" 200 None 2026-03-29 10:37:16,550 - lava_callback - ERROR - Error processing callback for node 69c65dd0b98d4d3c02068faa: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:34568 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50882 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46138 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37124 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48562 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48576 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37490 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:37500 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59182 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54938 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:38:36,532 - lava_callback - INFO - Received callback for node 69c65e1fb98d4d3c02068fb7 INFO: 172.21.0.1:54954 - "POST /node/69c65e1fb98d4d3c02068fb7 HTTP/1.0" 202 Accepted 2026-03-29 10:38:36,537 - lava_callback - INFO - Starting async job submit for node 69c65e1fb98d4d3c02068fb7 2026-03-29 10:38:36,538 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:38:36,552 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c65e1fb98d4d3c02068fb7 HTTP/1.1" 200 None 2026-03-29 10:38:36,553 - lava_callback - DEBUG - Processing node 69c65e1fb98d4d3c02068fb7: baseline-arm64-baylibre 2026-03-29 10:38:36,553 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c65e1fb98d4d3c02068fb7 2026-03-29 10:38:36,554 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:38:36,567 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c65e1fb98d4d3c02068fb7 HTTP/1.1" 200 None 2026-03-29 10:38:36,568 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:38:36,581 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69be5461448e074b3ffe163c HTTP/1.1" 200 None 2026-03-29 10:38:36,583 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:38:36,604 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c65e1fb98d4d3c02068fb7 HTTP/1.1" 200 None 2026-03-29 10:38:36,605 - lava_callback - ERROR - Error processing callback for node 69c65e1fb98d4d3c02068fb7: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:53566 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49202 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46870 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46882 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57610 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:57616 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38358 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53698 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60506 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41492 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:39:56,607 - lava_callback - INFO - Received callback for node 69c65e69b98d4d3c02068fcb INFO: 172.21.0.1:41496 - "POST /node/69c65e69b98d4d3c02068fcb HTTP/1.0" 202 Accepted 2026-03-29 10:39:56,613 - lava_callback - INFO - Starting async job submit for node 69c65e69b98d4d3c02068fcb 2026-03-29 10:39:56,613 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:39:56,628 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c65e69b98d4d3c02068fcb HTTP/1.1" 200 None 2026-03-29 10:39:56,629 - lava_callback - DEBUG - Processing node 69c65e69b98d4d3c02068fcb: baseline-arm64-baylibre 2026-03-29 10:39:56,629 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c65e69b98d4d3c02068fcb 2026-03-29 10:39:56,630 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:39:56,643 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c65e69b98d4d3c02068fcb HTTP/1.1" 200 None 2026-03-29 10:39:56,644 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:39:56,658 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69be5461448e074b3ffe163c HTTP/1.1" 200 None 2026-03-29 10:39:56,660 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:39:56,702 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c65e69b98d4d3c02068fcb HTTP/1.1" 200 None 2026-03-29 10:39:56,703 - lava_callback - ERROR - Error processing callback for node 69c65e69b98d4d3c02068fcb: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:37142 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37154 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46382 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:46398 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37502 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40118 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37836 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50180 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53876 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53880 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57656 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:57672 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38260 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35766 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52916 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48426 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48708 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48722 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57566 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:57572 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:42:16,739 - lava_callback - INFO - Received callback for node 69c65f06b98d4d3c02068fe3 INFO: 172.21.0.1:57578 - "POST /node/69c65f06b98d4d3c02068fe3 HTTP/1.0" 202 Accepted 2026-03-29 10:42:16,745 - lava_callback - INFO - Starting async job submit for node 69c65f06b98d4d3c02068fe3 2026-03-29 10:42:16,746 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:42:16,760 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c65f06b98d4d3c02068fe3 HTTP/1.1" 200 None 2026-03-29 10:42:16,761 - lava_callback - DEBUG - Processing node 69c65f06b98d4d3c02068fe3: baseline-arm64-baylibre 2026-03-29 10:42:16,761 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c65f06b98d4d3c02068fe3 2026-03-29 10:42:16,761 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:42:16,778 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c65f06b98d4d3c02068fe3 HTTP/1.1" 200 None 2026-03-29 10:42:16,780 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:42:16,794 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69be5461448e074b3ffe163c HTTP/1.1" 200 None 2026-03-29 10:42:16,795 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:42:16,816 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c65f06b98d4d3c02068fe3 HTTP/1.1" 200 None 2026-03-29 10:42:16,817 - lava_callback - ERROR - Error processing callback for node 69c65f06b98d4d3c02068fe3: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:50852 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51840 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47706 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46538 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36370 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36382 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39636 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:39644 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:43:20,346 - lava_callback - INFO - Received callback for node 69c9008f6a4c026e16348d80 INFO: 172.21.0.1:37058 - "POST /node/69c9008f6a4c026e16348d80 HTTP/1.0" 202 Accepted 2026-03-29 10:43:20,353 - lava_callback - INFO - Starting async job submit for node 69c9008f6a4c026e16348d80 2026-03-29 10:43:20,388 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:43:20,403 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9008f6a4c026e16348d80 HTTP/1.1" 200 None 2026-03-29 10:43:20,404 - lava_callback - DEBUG - Processing node 69c9008f6a4c026e16348d80: baseline-arm64 2026-03-29 10:43:20,790 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:43:20,986 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:43:20,987 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9008f6a4c026e16348d80/log.txt.gz 2026-03-29 10:43:21,202 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:43:21,434 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:43:21,436 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9008f6a4c026e16348d80/lava-logs.yaml 2026-03-29 10:43:21,445 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:43:21,770 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:43:21,771 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9008f6a4c026e16348d80/lava_callback.json.gz DEBUG: Setup failed for 69c9008f6a4c026e16348d80. Transitting job node result: incomplete -> fail 2026-03-29 10:43:21,844 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:43:21,867 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9008f6a4c026e16348d80 HTTP/1.1" 200 None 2026-03-29 10:43:21,869 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:43:21,883 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 10:43:21,884 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:43:21,908 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9008f6a4c026e16348d80 HTTP/1.1" 200 None 2026-03-29 10:43:21,989 - lava_callback - INFO - Completed processing callback for node 69c9008f6a4c026e16348d80 INFO: 172.21.0.1:37072 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58700 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:43:36,796 - lava_callback - INFO - Received callback for node 69c65f55b98d4d3c02068fee INFO: 172.21.0.1:58714 - "POST /node/69c65f55b98d4d3c02068fee HTTP/1.0" 202 Accepted 2026-03-29 10:43:36,803 - lava_callback - INFO - Starting async job submit for node 69c65f55b98d4d3c02068fee 2026-03-29 10:43:36,804 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:43:36,824 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c65f55b98d4d3c02068fee HTTP/1.1" 200 None 2026-03-29 10:43:36,824 - lava_callback - DEBUG - Processing node 69c65f55b98d4d3c02068fee: baseline-arm64-baylibre 2026-03-29 10:43:36,825 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c65f55b98d4d3c02068fee 2026-03-29 10:43:36,825 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:43:36,840 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c65f55b98d4d3c02068fee HTTP/1.1" 200 None 2026-03-29 10:43:36,841 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:43:36,863 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69be5461448e074b3ffe163c HTTP/1.1" 200 None 2026-03-29 10:43:36,864 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:43:36,886 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c65f55b98d4d3c02068fee HTTP/1.1" 200 None 2026-03-29 10:43:36,887 - lava_callback - ERROR - Error processing callback for node 69c65f55b98d4d3c02068fee: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-29 10:43:40,384 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:43:40,399 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:34824 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51206 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58778 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58792 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39608 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:39614 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34994 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55274 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41040 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37966 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35944 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35960 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:45:09,117 - lava_callback - INFO - Received callback for node 69c65fabb98d4d3c02069000 INFO: 172.21.0.1:35964 - "POST /node/69c65fabb98d4d3c02069000 HTTP/1.0" 202 Accepted 2026-03-29 10:45:09,123 - lava_callback - INFO - Starting async job submit for node 69c65fabb98d4d3c02069000 2026-03-29 10:45:09,124 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:45:09,138 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c65fabb98d4d3c02069000 HTTP/1.1" 200 None 2026-03-29 10:45:09,139 - lava_callback - DEBUG - Processing node 69c65fabb98d4d3c02069000: baseline-arm64-clabbe 2026-03-29 10:45:09,140 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c65fabb98d4d3c02069000 2026-03-29 10:45:09,140 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:45:09,158 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c65fabb98d4d3c02069000 HTTP/1.1" 200 None 2026-03-29 10:45:09,160 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:45:09,173 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69be5461448e074b3ffe163c HTTP/1.1" 200 None 2026-03-29 10:45:09,174 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:45:09,196 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c65fabb98d4d3c02069000 HTTP/1.1" 200 None 2026-03-29 10:45:09,198 - lava_callback - ERROR - Error processing callback for node 69c65fabb98d4d3c02069000: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:59360 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:59362 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51046 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43628 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37552 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42114 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47864 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47876 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58478 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:58486 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59600 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49220 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46814 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57374 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42536 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42542 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45532 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:45536 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35658 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47190 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36988 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:47:49,368 - lava_callback - INFO - Received callback for node 69c900da6a4c026e16348d8b INFO: 172.21.0.1:36998 - "POST /node/69c900da6a4c026e16348d8b HTTP/1.0" 202 Accepted 2026-03-29 10:47:49,375 - lava_callback - INFO - Starting async job submit for node 69c900da6a4c026e16348d8b 2026-03-29 10:47:49,399 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:47:49,414 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c900da6a4c026e16348d8b HTTP/1.1" 200 None 2026-03-29 10:47:49,415 - lava_callback - DEBUG - Processing node 69c900da6a4c026e16348d8b: baseline-arm64 2026-03-29 10:47:49,946 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:47:50,107 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:47:50,109 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c900da6a4c026e16348d8b/log.txt.gz 2026-03-29 10:47:50,357 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:47:50,602 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:47:50,604 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c900da6a4c026e16348d8b/lava-logs.yaml 2026-03-29 10:47:50,616 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:47:50,766 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:47:50,768 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c900da6a4c026e16348d8b/lava_callback.json.gz 2026-03-29 10:47:50,794 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:47:50,810 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c900da6a4c026e16348d8b HTTP/1.1" 200 None 2026-03-29 10:47:50,811 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:47:50,824 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 10:47:50,826 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:47:50,846 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c900da6a4c026e16348d8b HTTP/1.1" 200 None 2026-03-29 10:47:50,904 - lava_callback - INFO - Completed processing callback for node 69c900da6a4c026e16348d8b INFO: 172.21.0.1:51784 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58080 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58092 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:48:10,402 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:48:10,418 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:47756 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47768 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33116 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45230 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55762 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54086 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48366 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48376 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:49:09,212 - lava_callback - INFO - Received callback for node 69c6609bb98d4d3c0206903d INFO: 172.21.0.1:48380 - "POST /node/69c6609bb98d4d3c0206903d HTTP/1.0" 202 Accepted 2026-03-29 10:49:09,217 - lava_callback - INFO - Starting async job submit for node 69c6609bb98d4d3c0206903d 2026-03-29 10:49:09,218 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:49:09,233 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6609bb98d4d3c0206903d HTTP/1.1" 200 None 2026-03-29 10:49:09,234 - lava_callback - DEBUG - Processing node 69c6609bb98d4d3c0206903d: baseline-arm64-clabbe 2026-03-29 10:49:09,234 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c6609bb98d4d3c0206903d 2026-03-29 10:49:09,235 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:49:09,249 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6609bb98d4d3c0206903d HTTP/1.1" 200 None 2026-03-29 10:49:09,250 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:49:09,274 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69be5461448e074b3ffe163c HTTP/1.1" 200 None 2026-03-29 10:49:09,276 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:49:09,296 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c6609bb98d4d3c0206903d HTTP/1.1" 200 None 2026-03-29 10:49:09,301 - lava_callback - ERROR - Error processing callback for node 69c6609bb98d4d3c0206903d: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:59472 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:59474 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54728 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47530 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50300 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58550 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38182 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38186 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54324 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:54330 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44202 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:50:29,242 - lava_callback - INFO - Received callback for node 69c660efb98d4d3c02069054 INFO: 172.21.0.1:44216 - "POST /node/69c660efb98d4d3c02069054 HTTP/1.0" 202 Accepted 2026-03-29 10:50:29,247 - lava_callback - INFO - Starting async job submit for node 69c660efb98d4d3c02069054 2026-03-29 10:50:29,248 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:50:29,270 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c660efb98d4d3c02069054 HTTP/1.1" 200 None 2026-03-29 10:50:29,271 - lava_callback - DEBUG - Processing node 69c660efb98d4d3c02069054: baseline-arm64-clabbe 2026-03-29 10:50:29,271 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c660efb98d4d3c02069054 2026-03-29 10:50:29,272 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:50:29,286 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c660efb98d4d3c02069054 HTTP/1.1" 200 None 2026-03-29 10:50:29,288 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:50:29,305 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69be5461448e074b3ffe163c HTTP/1.1" 200 None 2026-03-29 10:50:29,307 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:50:29,328 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c660efb98d4d3c02069054 HTTP/1.1" 200 None 2026-03-29 10:50:29,330 - lava_callback - ERROR - Error processing callback for node 69c660efb98d4d3c02069054: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:43480 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52668 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56302 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38080 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38092 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55594 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:55610 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40708 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36450 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55620 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55736 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34622 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34636 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:52:09,362 - lava_callback - INFO - Received callback for node 69c66147b98d4d3c02069071 INFO: 172.21.0.1:34646 - "POST /node/69c66147b98d4d3c02069071 HTTP/1.0" 202 Accepted 2026-03-29 10:52:09,368 - lava_callback - INFO - Starting async job submit for node 69c66147b98d4d3c02069071 2026-03-29 10:52:09,369 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:52:09,383 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66147b98d4d3c02069071 HTTP/1.1" 200 None 2026-03-29 10:52:09,383 - lava_callback - DEBUG - Processing node 69c66147b98d4d3c02069071: baseline-arm64-clabbe 2026-03-29 10:52:09,384 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c66147b98d4d3c02069071 2026-03-29 10:52:09,384 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:52:09,398 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66147b98d4d3c02069071 HTTP/1.1" 200 None 2026-03-29 10:52:09,400 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:52:09,415 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69be5461448e074b3ffe163c HTTP/1.1" 200 None 2026-03-29 10:52:09,417 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:52:09,447 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c66147b98d4d3c02069071 HTTP/1.1" 200 None 2026-03-29 10:52:09,452 - lava_callback - ERROR - Error processing callback for node 69c66147b98d4d3c02069071: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:60022 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:60034 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38262 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55406 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53226 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48868 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60728 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60736 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37076 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:37080 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56312 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:53:29,455 - lava_callback - INFO - Received callback for node 69c66195b98d4d3c02069082 INFO: 172.21.0.1:56328 - "POST /node/69c66195b98d4d3c02069082 HTTP/1.0" 202 Accepted 2026-03-29 10:53:29,460 - lava_callback - INFO - Starting async job submit for node 69c66195b98d4d3c02069082 2026-03-29 10:53:29,461 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:53:29,489 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66195b98d4d3c02069082 HTTP/1.1" 200 None 2026-03-29 10:53:29,490 - lava_callback - DEBUG - Processing node 69c66195b98d4d3c02069082: baseline-arm64-clabbe 2026-03-29 10:53:29,490 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c66195b98d4d3c02069082 2026-03-29 10:53:29,491 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:53:29,507 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66195b98d4d3c02069082 HTTP/1.1" 200 None 2026-03-29 10:53:29,509 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:53:29,522 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69be5461448e074b3ffe163c HTTP/1.1" 200 None 2026-03-29 10:53:29,524 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:53:29,545 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c66195b98d4d3c02069082 HTTP/1.1" 200 None 2026-03-29 10:53:29,546 - lava_callback - ERROR - Error processing callback for node 69c66195b98d4d3c02069082: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:57146 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37268 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38240 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49612 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49618 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52382 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:52386 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35048 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:54:32,614 - lava_callback - INFO - Received callback for node 69c900436a4c026e16348d74 INFO: 172.21.0.1:42388 - "POST /node/69c900436a4c026e16348d74 HTTP/1.0" 202 Accepted 2026-03-29 10:54:32,621 - lava_callback - INFO - Starting async job submit for node 69c900436a4c026e16348d74 2026-03-29 10:54:32,665 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:54:32,679 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c900436a4c026e16348d74 HTTP/1.1" 200 None 2026-03-29 10:54:32,680 - lava_callback - DEBUG - Processing node 69c900436a4c026e16348d74: baseline-nfs-arm64 2026-03-29 10:54:33,688 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:54:33,885 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:54:33,887 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c900436a4c026e16348d74/log.txt.gz 2026-03-29 10:54:34,480 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:54:34,880 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:54:34,883 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c900436a4c026e16348d74/lava-logs.yaml 2026-03-29 10:54:34,903 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:54:35,068 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:54:35,070 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c900436a4c026e16348d74/lava_callback.json.gz DEBUG: Setup failed for 69c900436a4c026e16348d74. Transitting job node result: incomplete -> fail 2026-03-29 10:54:35,159 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:54:35,182 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c900436a4c026e16348d74 HTTP/1.1" 200 None 2026-03-29 10:54:35,184 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:54:35,197 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 10:54:35,199 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:54:35,233 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c900436a4c026e16348d74 HTTP/1.1" 200 None 2026-03-29 10:54:35,315 - lava_callback - INFO - Completed processing callback for node 69c900436a4c026e16348d74 INFO: 172.21.0.1:42392 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:54:40,421 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:54:40,437 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:53654 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60900 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53552 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53568 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48020 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48036 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54064 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47792 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57274 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39320 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:56:05,562 - lava_callback - INFO - Received callback for node 69c901276a4c026e16348d93 INFO: 172.21.0.1:47806 - "POST /node/69c901276a4c026e16348d93 HTTP/1.0" 202 Accepted 2026-03-29 10:56:05,568 - lava_callback - INFO - Starting async job submit for node 69c901276a4c026e16348d93 2026-03-29 10:56:05,626 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 INFO: 172.21.0.1:47812 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:56:05,641 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c901276a4c026e16348d93 HTTP/1.1" 200 None 2026-03-29 10:56:05,642 - lava_callback - DEBUG - Processing node 69c901276a4c026e16348d93: ltp-smoke 2026-03-29 10:56:06,146 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:56:06,294 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:56:06,296 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-smoke-69c901276a4c026e16348d93/log.txt.gz INFO: 172.21.0.1:47814 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:56:06,568 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:56:06,788 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:56:06,790 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-smoke-69c901276a4c026e16348d93/lava-logs.yaml 2026-03-29 10:56:06,801 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:56:06,971 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:56:06,973 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-smoke-69c901276a4c026e16348d93/lava_callback.json.gz 2026-03-29 10:56:07,091 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:56:07,106 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c901276a4c026e16348d93 HTTP/1.1" 200 None 2026-03-29 10:56:07,108 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:56:07,121 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff06a4c026e16347d3c HTTP/1.1" 200 None 2026-03-29 10:56:07,122 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:56:07,147 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c901276a4c026e16348d93 HTTP/1.1" 200 None 2026-03-29 10:56:07,277 - lava_callback - INFO - Completed processing callback for node 69c901276a4c026e16348d93 2026-03-29 10:56:10,439 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:56:10,465 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:37222 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:37226 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47716 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:56:29,535 - lava_callback - INFO - Received callback for node 69c66257b98d4d3c020690b1 INFO: 172.21.0.1:47718 - "POST /node/69c66257b98d4d3c020690b1 HTTP/1.0" 202 Accepted 2026-03-29 10:56:29,541 - lava_callback - INFO - Starting async job submit for node 69c66257b98d4d3c020690b1 2026-03-29 10:56:29,542 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:56:29,564 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66257b98d4d3c020690b1 HTTP/1.1" 200 None 2026-03-29 10:56:29,565 - lava_callback - DEBUG - Processing node 69c66257b98d4d3c020690b1: baseline-arm64-clabbe 2026-03-29 10:56:29,565 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c66257b98d4d3c020690b1 2026-03-29 10:56:29,566 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:56:29,590 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66257b98d4d3c020690b1 HTTP/1.1" 200 None 2026-03-29 10:56:29,591 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:56:29,635 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69be5461448e074b3ffe163c HTTP/1.1" 200 None 2026-03-29 10:56:29,636 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:56:29,656 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c66257b98d4d3c020690b1 HTTP/1.1" 200 None 2026-03-29 10:56:29,665 - lava_callback - ERROR - Error processing callback for node 69c66257b98d4d3c020690b1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:50624 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46676 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34930 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44958 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44968 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42622 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:42628 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36366 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49682 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35794 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:57:51,795 - lava_callback - INFO - Received callback for node 69c901746a4c026e16348da0 INFO: 172.21.0.1:47652 - "POST /node/69c901746a4c026e16348da0 HTTP/1.0" 202 Accepted 2026-03-29 10:57:51,802 - lava_callback - INFO - Starting async job submit for node 69c901746a4c026e16348da0 2026-03-29 10:57:51,861 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:57:51,878 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c901746a4c026e16348da0 HTTP/1.1" 200 None 2026-03-29 10:57:51,879 - lava_callback - DEBUG - Processing node 69c901746a4c026e16348da0: baseline-arm 2026-03-29 10:57:52,504 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:57:52,804 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:57:52,806 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69c901746a4c026e16348da0/log.txt.gz 2026-03-29 10:57:53,164 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:57:53,386 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:57:53,388 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69c901746a4c026e16348da0/lava-logs.yaml 2026-03-29 10:57:53,402 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 10:57:53,562 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 10:57:53,564 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69c901746a4c026e16348da0/lava_callback.json.gz DEBUG: Setup failed for 69c901746a4c026e16348da0. Transitting job node result: incomplete -> fail 2026-03-29 10:57:53,677 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:57:53,691 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c901746a4c026e16348da0 HTTP/1.1" 200 None 2026-03-29 10:57:53,693 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:57:53,715 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 10:57:53,716 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:57:53,745 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c901746a4c026e16348da0 HTTP/1.1" 200 None 2026-03-29 10:57:53,858 - lava_callback - INFO - Completed processing callback for node 69c901746a4c026e16348da0 INFO: 172.21.0.1:47656 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55974 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55986 - "GET / HTTP/1.0" 200 OK 2026-03-29 10:58:10,467 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 10:58:10,491 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:37114 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:37126 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44672 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59584 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55144 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48704 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53536 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53538 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54344 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:54350 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53402 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46678 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58218 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42514 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50470 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50484 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52174 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-29 11:00:15,914 - lava_callback - INFO - Received callback for node 69c901a26a4c026e16348dab INFO: 172.21.0.1:52178 - "POST /node/69c901a26a4c026e16348dab HTTP/1.0" 202 Accepted 2026-03-29 11:00:15,961 - lava_callback - INFO - Starting async job submit for node 69c901a26a4c026e16348dab 2026-03-29 11:00:15,985 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:00:16,003 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c901a26a4c026e16348dab HTTP/1.1" 200 None 2026-03-29 11:00:16,004 - lava_callback - DEBUG - Processing node 69c901a26a4c026e16348dab: baseline-arm-pengutronix 2026-03-29 11:00:16,118 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 11:00:16,270 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 11:00:16,271 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c901a26a4c026e16348dab/log.txt.gz 2026-03-29 11:00:16,329 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 11:00:16,502 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 11:00:16,504 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c901a26a4c026e16348dab/lava-logs.yaml 2026-03-29 11:00:16,507 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:52192 - "GET / HTTP/1.0" 200 OK 2026-03-29 11:00:16,642 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 11:00:16,643 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c901a26a4c026e16348dab/lava_callback.json.gz DEBUG: Setup failed for 69c901a26a4c026e16348dab. Transitting job node result: incomplete -> fail 2026-03-29 11:00:16,691 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:00:16,706 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c901a26a4c026e16348dab HTTP/1.1" 200 None 2026-03-29 11:00:16,707 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:00:16,721 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dfeb6a4c026e16347d36 HTTP/1.1" 200 None 2026-03-29 11:00:16,723 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:00:16,746 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c901a26a4c026e16348dab HTTP/1.1" 200 None 2026-03-29 11:00:16,821 - lava_callback - INFO - Completed processing callback for node 69c901a26a4c026e16348dab INFO: 172.21.0.1:50944 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55680 - "GET / HTTP/1.0" 200 OK 2026-03-29 11:00:40,493 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:00:40,508 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-29 11:00:40,619 - lava_callback - INFO - Received callback for node 69c901316a4c026e16348d9a INFO: 172.21.0.1:39650 - "POST /node/69c901316a4c026e16348d9a HTTP/1.0" 202 Accepted 2026-03-29 11:00:40,627 - lava_callback - INFO - Starting async job submit for node 69c901316a4c026e16348d9a 2026-03-29 11:00:40,678 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:00:40,696 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c901316a4c026e16348d9a HTTP/1.1" 200 None 2026-03-29 11:00:40,697 - lava_callback - DEBUG - Processing node 69c901316a4c026e16348d9a: baseline-nfs-arm64 2026-03-29 11:00:41,213 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 11:00:41,344 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 11:00:41,346 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c901316a4c026e16348d9a/log.txt.gz 2026-03-29 11:00:41,606 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 11:00:41,796 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 11:00:41,798 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c901316a4c026e16348d9a/lava-logs.yaml 2026-03-29 11:00:41,810 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 11:00:41,952 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 11:00:41,953 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c901316a4c026e16348d9a/lava_callback.json.gz 2026-03-29 11:00:41,980 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:00:41,995 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c901316a4c026e16348d9a HTTP/1.1" 200 None 2026-03-29 11:00:41,997 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:00:42,010 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 11:00:42,011 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:00:42,032 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c901316a4c026e16348d9a HTTP/1.1" 200 None 2026-03-29 11:00:42,098 - lava_callback - INFO - Completed processing callback for node 69c901316a4c026e16348d9a INFO: 172.21.0.1:39658 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51862 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44842 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44856 - "GET / HTTP/1.0" 200 OK 2026-03-29 11:01:10,510 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:01:10,526 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:33630 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:33640 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52180 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37946 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40816 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38026 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34924 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34926 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58880 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:58884 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52584 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36704 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41526 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38798 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34036 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34052 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34726 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:34740 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45916 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39030 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41954 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57040 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48750 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48766 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50846 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:50854 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42460 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42424 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56198 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42202 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57602 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57606 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57086 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:57100 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37902 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48646 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51388 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39914 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46642 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46658 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48390 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48402 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54730 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40262 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40598 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55900 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41306 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41312 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36862 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:36878 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35342 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44430 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34770 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49004 - "GET / HTTP/1.0" 200 OK 2026-03-29 11:07:57,983 - lava_callback - INFO - Received callback for node 69c66502b98d4d3c0206911d INFO: 172.21.0.1:49012 - "POST /node/69c66502b98d4d3c0206911d HTTP/1.0" 202 Accepted 2026-03-29 11:07:57,988 - lava_callback - INFO - Starting async job submit for node 69c66502b98d4d3c0206911d 2026-03-29 11:07:57,989 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:07:58,007 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66502b98d4d3c0206911d HTTP/1.1" 200 None 2026-03-29 11:07:58,008 - lava_callback - DEBUG - Processing node 69c66502b98d4d3c0206911d: baseline-arm64-baylibre 2026-03-29 11:07:58,008 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c66502b98d4d3c0206911d 2026-03-29 11:07:58,009 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:07:58,026 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66502b98d4d3c0206911d HTTP/1.1" 200 None 2026-03-29 11:07:58,028 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:07:58,042 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c5b114abf66a8f4d9bcb37 HTTP/1.1" 200 None 2026-03-29 11:07:58,044 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:07:58,065 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c66502b98d4d3c0206911d HTTP/1.1" 200 None 2026-03-29 11:07:58,067 - lava_callback - ERROR - Error processing callback for node 69c66502b98d4d3c0206911d: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:35590 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35596 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52394 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:52408 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49450 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57848 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34846 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53894 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53878 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53888 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42246 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:42256 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44834 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38176 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33340 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35642 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36172 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36188 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56368 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:56372 - "GET / HTTP/1.0" 200 OK 2026-03-29 11:10:18,124 - lava_callback - INFO - Received callback for node 69c66596b98d4d3c02069137 INFO: 172.21.0.1:56376 - "POST /node/69c66596b98d4d3c02069137 HTTP/1.0" 202 Accepted 2026-03-29 11:10:18,130 - lava_callback - INFO - Starting async job submit for node 69c66596b98d4d3c02069137 2026-03-29 11:10:18,131 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:10:18,149 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66596b98d4d3c02069137 HTTP/1.1" 200 None 2026-03-29 11:10:18,150 - lava_callback - DEBUG - Processing node 69c66596b98d4d3c02069137: baseline-arm64-baylibre 2026-03-29 11:10:18,150 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c66596b98d4d3c02069137 2026-03-29 11:10:18,151 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:10:18,167 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66596b98d4d3c02069137 HTTP/1.1" 200 None 2026-03-29 11:10:18,169 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:10:18,184 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c5b114abf66a8f4d9bcb37 HTTP/1.1" 200 None 2026-03-29 11:10:18,186 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:10:18,207 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c66596b98d4d3c02069137 HTTP/1.1" 200 None 2026-03-29 11:10:18,216 - lava_callback - ERROR - Error processing callback for node 69c66596b98d4d3c02069137: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:35974 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57658 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57410 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41180 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32798 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32802 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44684 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:44700 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45598 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34524 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54876 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53252 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40582 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40586 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48956 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48964 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58028 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42120 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50378 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60034 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52976 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52988 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46874 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:46878 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55582 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42440 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57512 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36550 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42124 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42130 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48362 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48378 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46112 - "GET / HTTP/1.0" 200 OK 2026-03-29 11:14:30,142 - lava_callback - INFO - Received callback for node 69c904f46a4c026e16348e39 INFO: 172.21.0.1:59112 - "POST /node/69c904f46a4c026e16348e39 HTTP/1.0" 202 Accepted 2026-03-29 11:14:30,150 - lava_callback - INFO - Starting async job submit for node 69c904f46a4c026e16348e39 2026-03-29 11:14:30,218 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:14:30,238 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c904f46a4c026e16348e39 HTTP/1.1" 200 None 2026-03-29 11:14:30,239 - lava_callback - DEBUG - Processing node 69c904f46a4c026e16348e39: baseline-nfs-arm64 2026-03-29 11:14:31,278 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 11:14:31,458 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 11:14:31,460 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c904f46a4c026e16348e39/log.txt.gz 2026-03-29 11:14:32,039 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 11:14:32,294 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 11:14:32,296 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c904f46a4c026e16348e39/lava-logs.yaml 2026-03-29 11:14:32,316 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 11:14:32,489 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 11:14:32,490 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c904f46a4c026e16348e39/lava_callback.json.gz DEBUG: Setup failed for 69c904f46a4c026e16348e39. Transitting job node result: incomplete -> fail 2026-03-29 11:14:32,588 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:14:32,604 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c904f46a4c026e16348e39 HTTP/1.1" 200 None 2026-03-29 11:14:32,606 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:14:32,645 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8dff86a4c026e16347d45 HTTP/1.1" 200 None 2026-03-29 11:14:32,647 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:14:32,674 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c904f46a4c026e16348e39 HTTP/1.1" 200 None 2026-03-29 11:14:32,772 - lava_callback - INFO - Completed processing callback for node 69c904f46a4c026e16348e39 INFO: 172.21.0.1:59128 - "GET / HTTP/1.0" 200 OK 2026-03-29 11:14:40,531 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:14:40,549 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:38108 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55990 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52776 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52786 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48986 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48998 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59464 - "GET / HTTP/1.0" 200 OK 2026-03-29 11:15:30,343 - lava_callback - INFO - Received callback for node 69c666c7b98d4d3c02069152 INFO: 172.21.0.1:45490 - "POST /node/69c666c7b98d4d3c02069152 HTTP/1.0" 202 Accepted 2026-03-29 11:15:30,349 - lava_callback - INFO - Starting async job submit for node 69c666c7b98d4d3c02069152 2026-03-29 11:15:30,349 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:15:30,371 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c666c7b98d4d3c02069152 HTTP/1.1" 200 None 2026-03-29 11:15:30,372 - lava_callback - DEBUG - Processing node 69c666c7b98d4d3c02069152: baseline-arm64-clabbe 2026-03-29 11:15:30,372 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c666c7b98d4d3c02069152 2026-03-29 11:15:30,373 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:15:30,396 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c666c7b98d4d3c02069152 HTTP/1.1" 200 None 2026-03-29 11:15:30,397 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:15:30,411 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c5b114abf66a8f4d9bcb37 HTTP/1.1" 200 None 2026-03-29 11:15:30,413 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:15:30,433 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c666c7b98d4d3c02069152 HTTP/1.1" 200 None 2026-03-29 11:15:30,439 - lava_callback - ERROR - Error processing callback for node 69c666c7b98d4d3c02069152: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:45494 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60338 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60056 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52400 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52414 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47356 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47370 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52296 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51618 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58182 - "GET / HTTP/1.0" 200 OK 2026-03-29 11:16:50,385 - lava_callback - INFO - Received callback for node 69c6670fb98d4d3c0206915f INFO: 172.21.0.1:45440 - "POST /node/69c6670fb98d4d3c0206915f HTTP/1.0" 202 Accepted 2026-03-29 11:16:50,391 - lava_callback - INFO - Starting async job submit for node 69c6670fb98d4d3c0206915f 2026-03-29 11:16:50,392 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:16:50,406 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6670fb98d4d3c0206915f HTTP/1.1" 200 None 2026-03-29 11:16:50,407 - lava_callback - DEBUG - Processing node 69c6670fb98d4d3c0206915f: baseline-arm64-clabbe 2026-03-29 11:16:50,407 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c6670fb98d4d3c0206915f 2026-03-29 11:16:50,408 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:16:50,425 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6670fb98d4d3c0206915f HTTP/1.1" 200 None 2026-03-29 11:16:50,426 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:16:50,439 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c5b114abf66a8f4d9bcb37 HTTP/1.1" 200 None 2026-03-29 11:16:50,441 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:16:50,461 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c6670fb98d4d3c0206915f HTTP/1.1" 200 None 2026-03-29 11:16:50,467 - lava_callback - ERROR - Error processing callback for node 69c6670fb98d4d3c0206915f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:45448 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41366 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41368 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39678 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:39690 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32792 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56662 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46760 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59510 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45970 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45974 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47350 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47364 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35996 - "GET / HTTP/1.0" 200 OK 2026-03-29 11:18:30,462 - lava_callback - INFO - Received callback for node 69c6677cb98d4d3c02069173 INFO: 172.21.0.1:51016 - "POST /node/69c6677cb98d4d3c02069173 HTTP/1.0" 202 Accepted 2026-03-29 11:18:30,467 - lava_callback - INFO - Starting async job submit for node 69c6677cb98d4d3c02069173 2026-03-29 11:18:30,469 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:18:30,483 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6677cb98d4d3c02069173 HTTP/1.1" 200 None 2026-03-29 11:18:30,484 - lava_callback - DEBUG - Processing node 69c6677cb98d4d3c02069173: baseline-arm64-clabbe 2026-03-29 11:18:30,484 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c6677cb98d4d3c02069173 2026-03-29 11:18:30,485 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:18:30,497 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6677cb98d4d3c02069173 HTTP/1.1" 200 None 2026-03-29 11:18:30,498 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:18:30,511 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c5b114abf66a8f4d9bcb37 HTTP/1.1" 200 None 2026-03-29 11:18:30,513 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:18:30,532 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c6677cb98d4d3c02069173 HTTP/1.1" 200 None 2026-03-29 11:18:30,541 - lava_callback - ERROR - Error processing callback for node 69c6677cb98d4d3c02069173: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:51026 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43846 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33878 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43238 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43250 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38842 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:38858 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49680 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46226 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36622 - "GET / HTTP/1.0" 200 OK 2026-03-29 11:19:50,540 - lava_callback - INFO - Received callback for node 69c667c5b98d4d3c0206917e INFO: 172.21.0.1:59712 - "POST /node/69c667c5b98d4d3c0206917e HTTP/1.0" 202 Accepted 2026-03-29 11:19:50,545 - lava_callback - INFO - Starting async job submit for node 69c667c5b98d4d3c0206917e 2026-03-29 11:19:50,546 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:19:50,560 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c667c5b98d4d3c0206917e HTTP/1.1" 200 None 2026-03-29 11:19:50,561 - lava_callback - DEBUG - Processing node 69c667c5b98d4d3c0206917e: baseline-arm64-clabbe 2026-03-29 11:19:50,561 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c667c5b98d4d3c0206917e 2026-03-29 11:19:50,562 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:19:50,584 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c667c5b98d4d3c0206917e HTTP/1.1" 200 None 2026-03-29 11:19:50,585 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:19:50,599 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c5b114abf66a8f4d9bcb37 HTTP/1.1" 200 None 2026-03-29 11:19:50,600 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:19:50,620 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c667c5b98d4d3c0206917e HTTP/1.1" 200 None 2026-03-29 11:19:50,621 - lava_callback - ERROR - Error processing callback for node 69c667c5b98d4d3c0206917e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:59724 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55274 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55286 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34498 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:34504 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33090 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58242 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33400 - "GET / HTTP/1.0" 200 OK 2026-03-29 11:20:50,571 - lava_callback - INFO - Received callback for node 69c6680eb98d4d3c02069191 INFO: 172.21.0.1:47442 - "POST /node/69c6680eb98d4d3c02069191 HTTP/1.0" 202 Accepted 2026-03-29 11:20:50,577 - lava_callback - INFO - Starting async job submit for node 69c6680eb98d4d3c02069191 2026-03-29 11:20:50,578 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:20:50,592 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6680eb98d4d3c02069191 HTTP/1.1" 200 None 2026-03-29 11:20:50,593 - lava_callback - DEBUG - Processing node 69c6680eb98d4d3c02069191: baseline-arm64-clabbe 2026-03-29 11:20:50,593 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c6680eb98d4d3c02069191 2026-03-29 11:20:50,594 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:20:50,607 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6680eb98d4d3c02069191 HTTP/1.1" 200 None 2026-03-29 11:20:50,609 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:20:50,646 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c5b114abf66a8f4d9bcb37 HTTP/1.1" 200 None 2026-03-29 11:20:50,647 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:20:50,668 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c6680eb98d4d3c02069191 HTTP/1.1" 200 None 2026-03-29 11:20:50,678 - lava_callback - ERROR - Error processing callback for node 69c6680eb98d4d3c02069191: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:47454 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47960 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47972 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43514 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:43524 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40648 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60874 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51530 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46250 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54774 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54784 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45982 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:45992 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46982 - "GET / HTTP/1.0" 200 OK 2026-03-29 11:22:30,643 - lava_callback - INFO - Received callback for node 69c6686cb98d4d3c020691a1 INFO: 172.21.0.1:39802 - "POST /node/69c6686cb98d4d3c020691a1 HTTP/1.0" 202 Accepted 2026-03-29 11:22:30,649 - lava_callback - INFO - Starting async job submit for node 69c6686cb98d4d3c020691a1 2026-03-29 11:22:30,649 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:22:30,664 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6686cb98d4d3c020691a1 HTTP/1.1" 200 None 2026-03-29 11:22:30,665 - lava_callback - DEBUG - Processing node 69c6686cb98d4d3c020691a1: baseline-arm64-clabbe 2026-03-29 11:22:30,666 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c6686cb98d4d3c020691a1 2026-03-29 11:22:30,666 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:22:30,688 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6686cb98d4d3c020691a1 HTTP/1.1" 200 None 2026-03-29 11:22:30,689 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:22:30,703 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c5b114abf66a8f4d9bcb37 HTTP/1.1" 200 None 2026-03-29 11:22:30,704 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:22:30,725 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c6686cb98d4d3c020691a1 HTTP/1.1" 200 None 2026-03-29 11:22:30,727 - lava_callback - ERROR - Error processing callback for node 69c6686cb98d4d3c020691a1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:39806 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32992 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37476 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34820 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34836 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53328 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:53336 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42958 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53606 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36218 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48242 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48276 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48282 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44528 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:44538 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47422 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59156 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59288 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39866 - "GET / HTTP/1.0" 200 OK 2026-03-29 11:24:59,243 - lava_callback - INFO - Received callback for node 69c668f5b98d4d3c020691bb INFO: 172.21.0.1:39882 - "POST /node/69c668f5b98d4d3c020691bb HTTP/1.0" 202 Accepted 2026-03-29 11:24:59,249 - lava_callback - INFO - Starting async job submit for node 69c668f5b98d4d3c020691bb 2026-03-29 11:24:59,250 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:24:59,264 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c668f5b98d4d3c020691bb HTTP/1.1" 200 None 2026-03-29 11:24:59,265 - lava_callback - DEBUG - Processing node 69c668f5b98d4d3c020691bb: baseline-arm-baylibre 2026-03-29 11:24:59,265 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c668f5b98d4d3c020691bb 2026-03-29 11:24:59,265 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:24:59,305 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c668f5b98d4d3c020691bb HTTP/1.1" 200 None 2026-03-29 11:24:59,306 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:24:59,327 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c55c70abf66a8f4d9b9655 HTTP/1.1" 200 None 2026-03-29 11:24:59,329 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:24:59,349 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c668f5b98d4d3c020691bb HTTP/1.1" 200 None 2026-03-29 11:24:59,350 - lava_callback - ERROR - Error processing callback for node 69c668f5b98d4d3c020691bb: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:55448 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55464 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36918 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:36934 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50672 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56950 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60276 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41708 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45674 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45690 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57496 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:57510 - "GET / HTTP/1.0" 200 OK 2026-03-29 11:26:19,013 - lava_callback - INFO - Received callback for node 69c6694ab98d4d3c020691c6 INFO: 172.21.0.1:57522 - "POST /node/69c6694ab98d4d3c020691c6 HTTP/1.0" 202 Accepted 2026-03-29 11:26:19,019 - lava_callback - INFO - Starting async job submit for node 69c6694ab98d4d3c020691c6 2026-03-29 11:26:19,020 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:26:19,040 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6694ab98d4d3c020691c6 HTTP/1.1" 200 None 2026-03-29 11:26:19,041 - lava_callback - DEBUG - Processing node 69c6694ab98d4d3c020691c6: baseline-arm-baylibre 2026-03-29 11:26:19,041 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c6694ab98d4d3c020691c6 2026-03-29 11:26:19,041 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:26:19,058 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6694ab98d4d3c020691c6 HTTP/1.1" 200 None 2026-03-29 11:26:19,060 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:26:19,073 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c55c70abf66a8f4d9b9655 HTTP/1.1" 200 None 2026-03-29 11:26:19,075 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:26:19,096 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c6694ab98d4d3c020691c6 HTTP/1.1" 200 None 2026-03-29 11:26:19,105 - lava_callback - ERROR - Error processing callback for node 69c6694ab98d4d3c020691c6: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:34084 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40790 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40094 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37396 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59214 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59224 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46810 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:46816 - "GET / HTTP/1.0" 200 OK 2026-03-29 11:27:19,028 - lava_callback - INFO - Received callback for node 69c66986b98d4d3c020691d2 INFO: 172.21.0.1:46826 - "POST /node/69c66986b98d4d3c020691d2 HTTP/1.0" 202 Accepted 2026-03-29 11:27:19,033 - lava_callback - INFO - Starting async job submit for node 69c66986b98d4d3c020691d2 2026-03-29 11:27:19,034 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:27:19,048 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66986b98d4d3c020691d2 HTTP/1.1" 200 None 2026-03-29 11:27:19,048 - lava_callback - DEBUG - Processing node 69c66986b98d4d3c020691d2: baseline-arm-baylibre 2026-03-29 11:27:19,049 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c66986b98d4d3c020691d2 2026-03-29 11:27:19,049 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:27:19,063 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66986b98d4d3c020691d2 HTTP/1.1" 200 None 2026-03-29 11:27:19,065 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:27:19,079 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c55c70abf66a8f4d9b9655 HTTP/1.1" 200 None 2026-03-29 11:27:19,080 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:27:19,115 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c66986b98d4d3c020691d2 HTTP/1.1" 200 None 2026-03-29 11:27:19,125 - lava_callback - ERROR - Error processing callback for node 69c66986b98d4d3c020691d2: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:49408 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41634 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35216 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57708 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47254 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47266 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43176 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:43180 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34690 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60198 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40234 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58188 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57326 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57334 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56262 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:56270 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35370 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44308 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59294 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35146 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40428 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40434 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58072 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:58074 - "GET / HTTP/1.0" 200 OK 2026-03-29 11:30:19,463 - lava_callback - INFO - Received callback for node 69c66a3fb98d4d3c020691f0 INFO: 172.21.0.1:58090 - "POST /node/69c66a3fb98d4d3c020691f0 HTTP/1.0" 202 Accepted 2026-03-29 11:30:19,468 - lava_callback - INFO - Starting async job submit for node 69c66a3fb98d4d3c020691f0 2026-03-29 11:30:19,469 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:30:19,490 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66a3fb98d4d3c020691f0 HTTP/1.1" 200 None 2026-03-29 11:30:19,491 - lava_callback - DEBUG - Processing node 69c66a3fb98d4d3c020691f0: baseline-arm-baylibre 2026-03-29 11:30:19,491 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c66a3fb98d4d3c020691f0 2026-03-29 11:30:19,491 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:30:19,505 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66a3fb98d4d3c020691f0 HTTP/1.1" 200 None 2026-03-29 11:30:19,506 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:30:19,520 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c55c70abf66a8f4d9b9655 HTTP/1.1" 200 None 2026-03-29 11:30:19,521 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:30:19,543 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c66a3fb98d4d3c020691f0 HTTP/1.1" 200 None 2026-03-29 11:30:19,549 - lava_callback - ERROR - Error processing callback for node 69c66a3fb98d4d3c020691f0: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:38642 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57548 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52148 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51538 - "GET / HTTP/1.0" 200 OK 2026-03-29 11:30:59,229 - lava_callback - INFO - Received callback for node 69c66a66b98d4d3c020691fc INFO: 172.21.0.1:51552 - "POST /node/69c66a66b98d4d3c020691fc HTTP/1.0" 202 Accepted 2026-03-29 11:30:59,234 - lava_callback - INFO - Starting async job submit for node 69c66a66b98d4d3c020691fc 2026-03-29 11:30:59,235 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:30:59,266 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66a66b98d4d3c020691fc HTTP/1.1" 200 None 2026-03-29 11:30:59,266 - lava_callback - DEBUG - Processing node 69c66a66b98d4d3c020691fc: baseline-arm-baylibre 2026-03-29 11:30:59,267 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c66a66b98d4d3c020691fc 2026-03-29 11:30:59,267 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:30:59,280 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66a66b98d4d3c020691fc HTTP/1.1" 200 None 2026-03-29 11:30:59,282 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:30:59,302 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c55c70abf66a8f4d9b9655 HTTP/1.1" 200 None 2026-03-29 11:30:59,303 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:30:59,324 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c66a66b98d4d3c020691fc HTTP/1.1" 200 None 2026-03-29 11:30:59,331 - lava_callback - ERROR - Error processing callback for node 69c66a66b98d4d3c020691fc: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:56664 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56670 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58944 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:58956 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50122 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56192 - "GET / HTTP/1.0" 200 OK 2026-03-29 11:31:39,315 - lava_callback - INFO - Received callback for node 69c66a8cb98d4d3c0206920c INFO: 172.21.0.1:56200 - "POST /node/69c66a8cb98d4d3c0206920c HTTP/1.0" 202 Accepted 2026-03-29 11:31:39,321 - lava_callback - INFO - Starting async job submit for node 69c66a8cb98d4d3c0206920c 2026-03-29 11:31:39,321 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:31:39,335 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66a8cb98d4d3c0206920c HTTP/1.1" 200 None 2026-03-29 11:31:39,336 - lava_callback - DEBUG - Processing node 69c66a8cb98d4d3c0206920c: baseline-arm-baylibre 2026-03-29 11:31:39,336 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c66a8cb98d4d3c0206920c 2026-03-29 11:31:39,337 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:31:39,350 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66a8cb98d4d3c0206920c HTTP/1.1" 200 None 2026-03-29 11:31:39,351 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:31:39,365 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c55c70abf66a8f4d9b9655 HTTP/1.1" 200 None 2026-03-29 11:31:39,366 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:31:39,388 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c66a8cb98d4d3c0206920c HTTP/1.1" 200 None 2026-03-29 11:31:39,390 - lava_callback - ERROR - Error processing callback for node 69c66a8cb98d4d3c0206920c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:44360 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36958 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35110 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35120 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33958 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:33970 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47214 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47318 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51022 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41746 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56788 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56804 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41362 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:41378 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51592 - "GET / HTTP/1.0" 200 OK 2026-03-29 11:33:31,133 - lava_callback - INFO - Received callback for node 69c66b04b98d4d3c02069237 INFO: 172.21.0.1:54200 - "POST /node/69c66b04b98d4d3c02069237 HTTP/1.0" 202 Accepted 2026-03-29 11:33:31,138 - lava_callback - INFO - Starting async job submit for node 69c66b04b98d4d3c02069237 2026-03-29 11:33:31,139 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:33:31,155 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66b04b98d4d3c02069237 HTTP/1.1" 200 None 2026-03-29 11:33:31,156 - lava_callback - DEBUG - Processing node 69c66b04b98d4d3c02069237: baseline-arm-clabbe 2026-03-29 11:33:31,156 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c66b04b98d4d3c02069237 2026-03-29 11:33:31,157 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:33:31,191 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66b04b98d4d3c02069237 HTTP/1.1" 200 None 2026-03-29 11:33:31,192 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:33:31,206 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c55c70abf66a8f4d9b9655 HTTP/1.1" 200 None 2026-03-29 11:33:31,207 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:33:31,238 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c66b04b98d4d3c02069237 HTTP/1.1" 200 None 2026-03-29 11:33:31,239 - lava_callback - ERROR - Error processing callback for node 69c66b04b98d4d3c02069237: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:54202 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48242 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60158 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33208 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33210 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34312 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:34324 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39258 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48998 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44748 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39536 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46400 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46412 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47002 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47006 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51822 - "GET / HTTP/1.0" 200 OK 2026-03-29 11:35:31,217 - lava_callback - INFO - Received callback for node 69c66b7fb98d4d3c02069250 INFO: 172.21.0.1:47060 - "POST /node/69c66b7fb98d4d3c02069250 HTTP/1.0" 202 Accepted 2026-03-29 11:35:31,223 - lava_callback - INFO - Starting async job submit for node 69c66b7fb98d4d3c02069250 2026-03-29 11:35:31,224 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:35:31,244 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66b7fb98d4d3c02069250 HTTP/1.1" 200 None 2026-03-29 11:35:31,245 - lava_callback - DEBUG - Processing node 69c66b7fb98d4d3c02069250: baseline-arm-clabbe 2026-03-29 11:35:31,245 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c66b7fb98d4d3c02069250 2026-03-29 11:35:31,246 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:35:31,259 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66b7fb98d4d3c02069250 HTTP/1.1" 200 None 2026-03-29 11:35:31,261 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:35:31,282 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c55c70abf66a8f4d9b9655 HTTP/1.1" 200 None 2026-03-29 11:35:31,283 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:35:31,304 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c66b7fb98d4d3c02069250 HTTP/1.1" 200 None 2026-03-29 11:35:31,309 - lava_callback - ERROR - Error processing callback for node 69c66b7fb98d4d3c02069250: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:47068 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55818 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46234 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59488 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59496 - "GET / HTTP/1.0" 200 OK 2026-03-29 11:36:11,207 - lava_callback - INFO - Received callback for node 69c66ba7b98d4d3c0206925b INFO: 172.21.0.1:42652 - "POST /node/69c66ba7b98d4d3c0206925b HTTP/1.0" 202 Accepted 2026-03-29 11:36:11,213 - lava_callback - INFO - Starting async job submit for node 69c66ba7b98d4d3c0206925b 2026-03-29 11:36:11,213 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:36:11,231 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66ba7b98d4d3c0206925b HTTP/1.1" 200 None 2026-03-29 11:36:11,232 - lava_callback - DEBUG - Processing node 69c66ba7b98d4d3c0206925b: baseline-arm-clabbe 2026-03-29 11:36:11,232 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c66ba7b98d4d3c0206925b 2026-03-29 11:36:11,233 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:36:11,246 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66ba7b98d4d3c0206925b HTTP/1.1" 200 None 2026-03-29 11:36:11,247 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:36:11,260 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c55c70abf66a8f4d9b9655 HTTP/1.1" 200 None 2026-03-29 11:36:11,261 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:36:11,281 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c66ba7b98d4d3c0206925b HTTP/1.1" 200 None 2026-03-29 11:36:11,291 - lava_callback - ERROR - Error processing callback for node 69c66ba7b98d4d3c0206925b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:42664 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:42680 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58176 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42924 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59954 - "GET / HTTP/1.0" 200 OK 2026-03-29 11:36:51,238 - lava_callback - INFO - Received callback for node 69c66bd0b98d4d3c02069268 INFO: 172.21.0.1:43426 - "POST /node/69c66bd0b98d4d3c02069268 HTTP/1.0" 202 Accepted 2026-03-29 11:36:51,243 - lava_callback - INFO - Starting async job submit for node 69c66bd0b98d4d3c02069268 2026-03-29 11:36:51,244 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:36:51,259 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66bd0b98d4d3c02069268 HTTP/1.1" 200 None 2026-03-29 11:36:51,259 - lava_callback - DEBUG - Processing node 69c66bd0b98d4d3c02069268: baseline-arm-clabbe 2026-03-29 11:36:51,260 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c66bd0b98d4d3c02069268 2026-03-29 11:36:51,260 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:36:51,273 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66bd0b98d4d3c02069268 HTTP/1.1" 200 None 2026-03-29 11:36:51,275 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:36:51,289 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c55c70abf66a8f4d9b9655 HTTP/1.1" 200 None 2026-03-29 11:36:51,290 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:36:51,311 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c66bd0b98d4d3c02069268 HTTP/1.1" 200 None 2026-03-29 11:36:51,316 - lava_callback - ERROR - Error processing callback for node 69c66bd0b98d4d3c02069268: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:43428 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54700 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54708 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39316 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:39324 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37026 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36754 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58796 - "GET / HTTP/1.0" 200 OK 2026-03-29 11:37:51,278 - lava_callback - INFO - Received callback for node 69c66bfab98d4d3c02069273 INFO: 172.21.0.1:45980 - "POST /node/69c66bfab98d4d3c02069273 HTTP/1.0" 202 Accepted 2026-03-29 11:37:51,284 - lava_callback - INFO - Starting async job submit for node 69c66bfab98d4d3c02069273 2026-03-29 11:37:51,285 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:37:51,299 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66bfab98d4d3c02069273 HTTP/1.1" 200 None 2026-03-29 11:37:51,300 - lava_callback - DEBUG - Processing node 69c66bfab98d4d3c02069273: baseline-arm-clabbe 2026-03-29 11:37:51,300 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c66bfab98d4d3c02069273 2026-03-29 11:37:51,301 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:37:51,314 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c66bfab98d4d3c02069273 HTTP/1.1" 200 None 2026-03-29 11:37:51,315 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:37:51,329 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c55c70abf66a8f4d9b9655 HTTP/1.1" 200 None 2026-03-29 11:37:51,330 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 11:37:51,356 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c66bfab98d4d3c02069273 HTTP/1.1" 200 None 2026-03-29 11:37:51,357 - lava_callback - ERROR - Error processing callback for node 69c66bfab98d4d3c02069273: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:45988 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47786 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47802 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60570 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:60572 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56642 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37488 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50876 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34830 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58466 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58476 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46496 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:46498 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40624 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51368 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39042 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34904 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45960 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45972 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50412 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:50414 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55666 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41306 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47082 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50272 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33726 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33742 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38882 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:38890 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48514 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46606 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59636 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58206 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37022 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37038 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39190 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:39202 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44544 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53250 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49822 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54614 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43728 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43740 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50276 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:50292 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54724 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33966 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58338 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33016 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60622 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60636 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33702 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:33704 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59544 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49180 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58784 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39672 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56328 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56336 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54606 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:54610 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50182 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54208 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35440 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51776 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45276 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45290 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42294 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:42310 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53330 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41924 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45084 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50132 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47848 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47852 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36588 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:36590 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52944 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60360 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50486 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51624 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42216 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42226 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44702 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:44712 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37908 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51320 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45798 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44338 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37484 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37500 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54848 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:54862 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38138 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55996 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40606 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49758 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44568 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44584 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46914 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:46924 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34192 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51258 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41378 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40934 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44306 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44316 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37294 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:37310 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40896 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48274 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60214 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49954 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45056 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45060 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36832 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:36842 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41084 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33744 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46054 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38478 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41634 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41648 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57464 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:57466 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44048 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54250 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39228 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48868 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35814 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35828 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48280 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48296 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53060 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45070 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60496 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47818 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45544 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45552 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33642 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:33650 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33332 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57784 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48556 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53428 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39554 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39556 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57736 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:57746 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47962 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47686 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46382 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34472 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59698 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59712 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46280 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:46284 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42322 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42932 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38644 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45570 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51300 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51306 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50730 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:50744 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42994 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60528 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35808 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59378 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33708 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33716 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46952 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:46958 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54552 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53998 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41314 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56754 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49696 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49698 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51844 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:51846 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52304 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57948 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39990 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34322 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52644 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52658 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54598 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:54604 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57034 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39176 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37144 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37172 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44896 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44898 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35778 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:35792 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41782 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35506 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39016 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35850 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60840 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60842 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55842 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:55848 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52946 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41028 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50124 - "GET / HTTP/1.0" 200 OK 2026-03-29 12:03:52,739 - lava_callback - INFO - Received callback for node 69c8e4696a4c026e1634837a INFO: 172.21.0.1:43304 - "POST /node/69c8e4696a4c026e1634837a HTTP/1.0" 202 Accepted 2026-03-29 12:03:52,745 - lava_callback - INFO - Starting async job submit for node 69c8e4696a4c026e1634837a 2026-03-29 12:03:52,762 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 12:03:52,793 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4696a4c026e1634837a HTTP/1.1" 200 None 2026-03-29 12:03:52,794 - lava_callback - DEBUG - Processing node 69c8e4696a4c026e1634837a: baseline-x86 2026-03-29 12:03:53,089 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 12:03:53,263 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 12:03:53,265 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-x86-69c8e4696a4c026e1634837a/log.txt.gz 2026-03-29 12:03:53,424 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 12:03:53,653 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 12:03:53,655 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69c8e4696a4c026e1634837a/lava-logs.yaml 2026-03-29 12:03:53,662 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-29 12:03:53,818 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-29 12:03:53,819 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69c8e4696a4c026e1634837a/lava_callback.json.gz 2026-03-29 12:03:53,837 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 12:03:53,851 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e4696a4c026e1634837a HTTP/1.1" 200 None 2026-03-29 12:03:53,853 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 12:03:53,875 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c8e00a6a4c026e16347d61 HTTP/1.1" 200 None 2026-03-29 12:03:53,877 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 12:03:53,911 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c8e4696a4c026e1634837a HTTP/1.1" 200 None 2026-03-29 12:03:53,922 - lava_callback - INFO - Completed processing callback for node 69c8e4696a4c026e1634837a INFO: 172.21.0.1:43308 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37226 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37240 - "GET / HTTP/1.0" 200 OK 2026-03-29 12:04:10,560 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 12:04:10,583 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:41688 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:41696 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41934 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42496 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56578 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46606 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37426 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37442 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48522 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48536 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43652 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41660 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58640 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44488 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60128 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60144 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49408 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:49422 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42064 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37800 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32916 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37018 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40724 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40734 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41108 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:41116 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51248 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47642 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56328 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53542 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57262 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57278 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53598 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:53608 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57952 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36406 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59672 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53772 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38452 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38456 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60172 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:60174 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58702 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47206 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36782 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51962 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59660 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59676 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51380 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:51392 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60848 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46708 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36310 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47384 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50514 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50520 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58860 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:58872 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33070 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47996 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37674 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58920 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57166 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57180 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40528 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:40530 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56612 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47530 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37920 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48652 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52908 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52910 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44686 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:44700 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50896 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51496 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46692 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50718 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33198 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33210 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40412 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:40422 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39026 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46618 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37200 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51806 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59406 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59410 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40530 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:40544 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33704 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41248 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46702 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41956 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58786 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58798 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40360 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:40370 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47522 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58836 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56960 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34292 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56922 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56936 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56574 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:56590 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39668 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55160 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45770 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36774 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45460 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45468 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39050 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:39058 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44646 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51964 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46910 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33884 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33732 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33746 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51706 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:51720 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35366 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34500 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48238 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52280 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40428 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40436 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53664 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:53672 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42728 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37276 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45708 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57212 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56152 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56160 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60902 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:60912 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51678 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35134 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50906 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34076 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36068 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36072 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38276 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:38286 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51926 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38540 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34626 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58228 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54756 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54758 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57660 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:57664 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53728 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56414 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55268 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33642 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41944 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41956 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40768 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:40770 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55560 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44886 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51660 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35434 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39548 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39550 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56600 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:56602 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53220 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42822 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41882 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35906 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34898 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34912 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33202 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:33210 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60256 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41384 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46960 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59728 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57428 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57438 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38452 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:38464 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49286 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40666 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45334 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50908 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43664 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43672 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42204 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:42214 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57258 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56568 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57246 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46092 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39390 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39396 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46702 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:46706 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60730 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46066 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36430 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51642 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39198 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39214 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44902 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:44912 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53466 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41456 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54488 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39926 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58008 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58018 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47668 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47682 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47344 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60394 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49598 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38168 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53514 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53528 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34004 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:34008 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56908 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59670 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50100 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41914 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45562 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45576 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48084 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48092 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56558 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46858 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52816 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49526 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59874 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59876 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42068 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:42080 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51268 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58736 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41012 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56428 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34806 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34812 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44274 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:44280 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48054 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34274 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40082 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34636 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45948 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45958 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34738 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:34740 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51338 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58410 - "GET / HTTP/1.0" 200 OK 2026-03-29 12:36:43,094 - lava_callback - INFO - Received callback for node 69c679ccb98d4d3c02069314 INFO: 172.21.0.1:55994 - "POST /node/69c679ccb98d4d3c02069314 HTTP/1.0" 202 Accepted 2026-03-29 12:36:43,100 - lava_callback - INFO - Starting async job submit for node 69c679ccb98d4d3c02069314 2026-03-29 12:36:43,101 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 12:36:43,114 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c679ccb98d4d3c02069314 HTTP/1.1" 200 None 2026-03-29 12:36:43,115 - lava_callback - DEBUG - Processing node 69c679ccb98d4d3c02069314: baseline-arm-baylibre 2026-03-29 12:36:43,115 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c679ccb98d4d3c02069314 2026-03-29 12:36:43,116 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 12:36:43,129 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c679ccb98d4d3c02069314 HTTP/1.1" 200 None 2026-03-29 12:36:43,130 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 12:36:43,143 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c5cc292cb62322ad965da7 HTTP/1.1" 200 None 2026-03-29 12:36:43,145 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 12:36:43,165 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c679ccb98d4d3c02069314 HTTP/1.1" 200 None 2026-03-29 12:36:43,173 - lava_callback - ERROR - Error processing callback for node 69c679ccb98d4d3c02069314: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:56010 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36214 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39140 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39156 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55682 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:55694 - "GET / HTTP/1.0" 200 OK 2026-03-29 12:37:22,876 - lava_callback - INFO - Received callback for node 69c679f0b98d4d3c0206931f INFO: 172.21.0.1:40710 - "POST /node/69c679f0b98d4d3c0206931f HTTP/1.0" 202 Accepted 2026-03-29 12:37:22,881 - lava_callback - INFO - Starting async job submit for node 69c679f0b98d4d3c0206931f 2026-03-29 12:37:22,882 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 12:37:22,897 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c679f0b98d4d3c0206931f HTTP/1.1" 200 None 2026-03-29 12:37:22,898 - lava_callback - DEBUG - Processing node 69c679f0b98d4d3c0206931f: baseline-arm-baylibre 2026-03-29 12:37:22,898 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c679f0b98d4d3c0206931f 2026-03-29 12:37:22,899 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 12:37:22,911 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c679f0b98d4d3c0206931f HTTP/1.1" 200 None 2026-03-29 12:37:22,912 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 12:37:22,929 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c5cc292cb62322ad965da7 HTTP/1.1" 200 None 2026-03-29 12:37:22,931 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 12:37:22,951 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c679f0b98d4d3c0206931f HTTP/1.1" 200 None 2026-03-29 12:37:22,953 - lava_callback - ERROR - Error processing callback for node 69c679f0b98d4d3c0206931f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:40720 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41634 - "GET / HTTP/1.0" 200 OK 2026-03-29 12:37:42,894 - lava_callback - INFO - Received callback for node 69c67a14b98d4d3c0206932c INFO: 172.21.0.1:34100 - "POST /node/69c67a14b98d4d3c0206932c HTTP/1.0" 202 Accepted 2026-03-29 12:37:42,900 - lava_callback - INFO - Starting async job submit for node 69c67a14b98d4d3c0206932c 2026-03-29 12:37:42,900 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-29 12:37:42,915 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c67a14b98d4d3c0206932c HTTP/1.1" 200 None 2026-03-29 12:37:42,916 - lava_callback - DEBUG - Processing node 69c67a14b98d4d3c0206932c: baseline-arm-baylibre 2026-03-29 12:37:42,916 - lava_callback - WARNING - No log data found in callback