INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) INFO: 172.20.0.1:35064 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34448 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47050 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37388 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37394 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37398 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44462 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56328 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60448 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49166 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60434 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41506 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41518 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56944 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56950 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58586 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40086 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41894 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40772 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44490 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44500 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53228 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53244 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52694 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55760 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55634 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57496 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33110 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33122 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56748 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56762 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53186 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46440 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35256 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49760 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40908 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40910 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51216 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51218 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43582 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56650 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39932 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42698 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58628 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58642 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45234 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45236 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60478 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56250 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53352 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45826 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60516 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60524 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47544 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47550 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34764 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60968 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57940 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50960 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45588 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45592 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44072 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44078 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51738 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55552 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58688 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45670 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57914 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57924 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58444 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58448 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45392 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47168 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34442 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42816 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33302 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33318 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49546 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49560 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38178 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44934 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34962 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36454 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38316 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:38324 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57784 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57790 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49908 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:24:44,057 - lava_callback - INFO - Received callback for node 69e420c535519646173362c5 2026-04-21 00:24:44,063 - lava_callback - INFO - Starting async job submit for node 69e420c535519646173362c5 INFO: 172.20.0.1:52218 - "POST /node/69e420c535519646173362c5 HTTP/1.0" 202 Accepted 2026-04-21 00:24:44,065 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:24:44,084 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e420c535519646173362c5 HTTP/1.1" 200 None 2026-04-21 00:24:44,085 - lava_callback - DEBUG - Processing node 69e420c535519646173362c5: baseline-arm-baylibre 2026-04-21 00:24:44,086 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e420c535519646173362c5 2026-04-21 00:24:44,087 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:24:44,099 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e420c535519646173362c5 HTTP/1.1" 200 None 2026-04-21 00:24:44,100 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:24:44,114 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 00:24:44,115 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:24:44,143 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e420c535519646173362c5 HTTP/1.1" 200 None 2026-04-21 00:24:44,145 - lava_callback - ERROR - Error processing callback for node 69e420c535519646173362c5: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:52224 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55392 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58568 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33798 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33814 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45712 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:25:23,857 - lava_callback - INFO - Received callback for node 69e420eb35519646173362d2 INFO: 172.20.0.1:45728 - "POST /node/69e420eb35519646173362d2 HTTP/1.0" 202 Accepted 2026-04-21 00:25:23,863 - lava_callback - INFO - Starting async job submit for node 69e420eb35519646173362d2 2026-04-21 00:25:23,864 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:25:23,884 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e420eb35519646173362d2 HTTP/1.1" 200 None 2026-04-21 00:25:23,885 - lava_callback - DEBUG - Processing node 69e420eb35519646173362d2: baseline-arm-baylibre 2026-04-21 00:25:23,885 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e420eb35519646173362d2 2026-04-21 00:25:23,885 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:25:23,900 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e420eb35519646173362d2 HTTP/1.1" 200 None 2026-04-21 00:25:23,902 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:25:23,917 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 00:25:23,919 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:25:23,943 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e420eb35519646173362d2 HTTP/1.1" 200 None 2026-04-21 00:25:23,945 - lava_callback - ERROR - Error processing callback for node 69e420eb35519646173362d2: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:45740 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57530 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35514 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58940 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:26:03,928 - lava_callback - INFO - Received callback for node 69e4210f35519646173362e2 INFO: 172.20.0.1:60504 - "POST /node/69e4210f35519646173362e2 HTTP/1.0" 202 Accepted 2026-04-21 00:26:03,934 - lava_callback - INFO - Starting async job submit for node 69e4210f35519646173362e2 2026-04-21 00:26:03,935 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:26:03,949 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4210f35519646173362e2 HTTP/1.1" 200 None 2026-04-21 00:26:03,950 - lava_callback - DEBUG - Processing node 69e4210f35519646173362e2: baseline-arm-baylibre 2026-04-21 00:26:03,950 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4210f35519646173362e2 2026-04-21 00:26:03,951 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:26:03,970 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4210f35519646173362e2 HTTP/1.1" 200 None 2026-04-21 00:26:03,972 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:26:03,985 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 00:26:03,987 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:26:04,004 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4210f35519646173362e2 HTTP/1.1" 200 None 2026-04-21 00:26:04,006 - lava_callback - ERROR - Error processing callback for node 69e4210f35519646173362e2: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:60512 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53370 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53386 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56250 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56266 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56886 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:26:43,905 - lava_callback - INFO - Received callback for node 69e421343551964617336ac2 INFO: 172.20.0.1:48464 - "POST /node/69e421343551964617336ac2 HTTP/1.0" 202 Accepted 2026-04-21 00:26:43,910 - lava_callback - INFO - Starting async job submit for node 69e421343551964617336ac2 2026-04-21 00:26:43,912 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:26:43,930 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e421343551964617336ac2 HTTP/1.1" 200 None 2026-04-21 00:26:43,932 - lava_callback - DEBUG - Processing node 69e421343551964617336ac2: baseline-arm-baylibre 2026-04-21 00:26:43,932 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e421343551964617336ac2 2026-04-21 00:26:43,932 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:26:43,952 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e421343551964617336ac2 HTTP/1.1" 200 None 2026-04-21 00:26:43,954 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:26:43,971 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 00:26:43,973 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:26:43,992 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e421343551964617336ac2 HTTP/1.1" 200 None 2026-04-21 00:26:43,994 - lava_callback - ERROR - Error processing callback for node 69e421343551964617336ac2: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:48472 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42188 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55708 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51302 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51310 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45922 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:27:24,030 - lava_callback - INFO - Received callback for node 69e4215935519646173389d7 INFO: 172.20.0.1:45926 - "POST /node/69e4215935519646173389d7 HTTP/1.0" 202 Accepted 2026-04-21 00:27:24,035 - lava_callback - INFO - Starting async job submit for node 69e4215935519646173389d7 2026-04-21 00:27:24,036 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:27:24,053 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4215935519646173389d7 HTTP/1.1" 200 None 2026-04-21 00:27:24,054 - lava_callback - DEBUG - Processing node 69e4215935519646173389d7: baseline-arm-baylibre 2026-04-21 00:27:24,054 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4215935519646173389d7 2026-04-21 00:27:24,054 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:27:24,076 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4215935519646173389d7 HTTP/1.1" 200 None 2026-04-21 00:27:24,077 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:27:24,090 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 00:27:24,091 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:27:24,108 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4215935519646173389d7 HTTP/1.1" 200 None 2026-04-21 00:27:24,109 - lava_callback - ERROR - Error processing callback for node 69e4215935519646173389d7: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:45930 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59082 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55032 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51750 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:28:03,994 - lava_callback - INFO - Received callback for node 69e4217e35519646173389e5 INFO: 172.20.0.1:47874 - "POST /node/69e4217e35519646173389e5 HTTP/1.0" 202 Accepted 2026-04-21 00:28:04,000 - lava_callback - INFO - Starting async job submit for node 69e4217e35519646173389e5 2026-04-21 00:28:04,000 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:28:04,018 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4217e35519646173389e5 HTTP/1.1" 200 None 2026-04-21 00:28:04,019 - lava_callback - DEBUG - Processing node 69e4217e35519646173389e5: baseline-arm-baylibre 2026-04-21 00:28:04,019 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4217e35519646173389e5 2026-04-21 00:28:04,020 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:28:04,053 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4217e35519646173389e5 HTTP/1.1" 200 None 2026-04-21 00:28:04,055 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:28:04,069 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 00:28:04,070 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:28:04,112 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4217e35519646173389e5 HTTP/1.1" 200 None 2026-04-21 00:28:04,113 - lava_callback - ERROR - Error processing callback for node 69e4217e35519646173389e5: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47882 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41300 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41312 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35234 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35248 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:28:32,212 - lava_callback - INFO - Received callback for node 69e421a435519646173389f3 INFO: 172.20.0.1:38372 - "POST /node/69e421a435519646173389f3 HTTP/1.0" 202 Accepted 2026-04-21 00:28:32,218 - lava_callback - INFO - Starting async job submit for node 69e421a435519646173389f3 2026-04-21 00:28:32,219 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:28:32,237 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e421a435519646173389f3 HTTP/1.1" 200 None 2026-04-21 00:28:32,238 - lava_callback - DEBUG - Processing node 69e421a435519646173389f3: baseline-arm-clabbe 2026-04-21 00:28:32,238 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e421a435519646173389f3 2026-04-21 00:28:32,239 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:28:32,267 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e421a435519646173389f3 HTTP/1.1" 200 None 2026-04-21 00:28:32,269 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:28:32,288 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 00:28:32,290 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:28:32,307 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e421a435519646173389f3 HTTP/1.1" 200 None 2026-04-21 00:28:32,309 - lava_callback - ERROR - Error processing callback for node 69e421a435519646173389f3: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:38388 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58878 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58368 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:29:02,499 - lava_callback - INFO - Received callback for node 69e6c45960f5826562c15185 INFO: 172.20.0.1:43446 - "POST /node/69e6c45960f5826562c15185 HTTP/1.0" 202 Accepted 2026-04-21 00:29:02,505 - lava_callback - INFO - Starting async job submit for node 69e6c45960f5826562c15185 2026-04-21 00:29:02,528 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:29:02,576 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c45960f5826562c15185 HTTP/1.1" 200 None 2026-04-21 00:29:02,577 - lava_callback - DEBUG - Processing node 69e6c45960f5826562c15185: baseline-arm 2026-04-21 00:29:02,762 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:29:03,073 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:29:03,075 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69e6c45960f5826562c15185/log.txt.gz 2026-04-21 00:29:03,178 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:29:03,375 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:29:03,377 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69e6c45960f5826562c15185/lava-logs.yaml 2026-04-21 00:29:03,383 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:29:03,541 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:29:03,544 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69e6c45960f5826562c15185/lava_callback.json.gz 2026-04-21 00:29:03,570 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:29:03,614 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c45960f5826562c15185 HTTP/1.1" 200 None 2026-04-21 00:29:03,616 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:29:03,647 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c21560f5826562c15091 HTTP/1.1" 200 None 2026-04-21 00:29:03,648 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:29:03,670 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c45960f5826562c15185 HTTP/1.1" 200 None 2026-04-21 00:29:03,671 - lava_callback - INFO - Completed processing callback for node 69e6c45960f5826562c15185 INFO: 172.20.0.1:43456 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:29:12,051 - lava_callback - INFO - Received callback for node 69e421c93551964617338a03 INFO: 172.20.0.1:57388 - "POST /node/69e421c93551964617338a03 HTTP/1.0" 202 Accepted 2026-04-21 00:29:12,057 - lava_callback - INFO - Starting async job submit for node 69e421c93551964617338a03 2026-04-21 00:29:12,058 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:29:12,073 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e421c93551964617338a03 HTTP/1.1" 200 None 2026-04-21 00:29:12,074 - lava_callback - DEBUG - Processing node 69e421c93551964617338a03: baseline-arm-clabbe 2026-04-21 00:29:12,074 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e421c93551964617338a03 2026-04-21 00:29:12,075 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:29:12,087 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e421c93551964617338a03 HTTP/1.1" 200 None 2026-04-21 00:29:12,089 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:29:12,101 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 00:29:12,103 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:29:12,120 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e421c93551964617338a03 HTTP/1.1" 200 None 2026-04-21 00:29:12,121 - lava_callback - ERROR - Error processing callback for node 69e421c93551964617338a03: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-04-21 00:29:14,146 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:29:14,164 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:57402 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57412 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34340 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34356 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58558 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51402 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:29:52,056 - lava_callback - INFO - Received callback for node 69e421ee3551964617338a14 INFO: 172.20.0.1:38200 - "POST /node/69e421ee3551964617338a14 HTTP/1.0" 202 Accepted 2026-04-21 00:29:52,061 - lava_callback - INFO - Starting async job submit for node 69e421ee3551964617338a14 2026-04-21 00:29:52,062 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:29:52,076 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e421ee3551964617338a14 HTTP/1.1" 200 None 2026-04-21 00:29:52,077 - lava_callback - DEBUG - Processing node 69e421ee3551964617338a14: baseline-arm-clabbe 2026-04-21 00:29:52,077 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e421ee3551964617338a14 2026-04-21 00:29:52,078 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:29:52,090 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e421ee3551964617338a14 HTTP/1.1" 200 None 2026-04-21 00:29:52,092 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:29:52,113 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 00:29:52,115 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:29:52,132 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e421ee3551964617338a14 HTTP/1.1" 200 None 2026-04-21 00:29:52,134 - lava_callback - ERROR - Error processing callback for node 69e421ee3551964617338a14: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:38210 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39494 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52088 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52090 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33362 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33370 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:30:32,036 - lava_callback - INFO - Received callback for node 69e422133551964617338a1f INFO: 172.20.0.1:34164 - "POST /node/69e422133551964617338a1f HTTP/1.0" 202 Accepted 2026-04-21 00:30:32,042 - lava_callback - INFO - Starting async job submit for node 69e422133551964617338a1f 2026-04-21 00:30:32,043 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:30:32,057 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e422133551964617338a1f HTTP/1.1" 200 None 2026-04-21 00:30:32,058 - lava_callback - DEBUG - Processing node 69e422133551964617338a1f: baseline-arm-clabbe 2026-04-21 00:30:32,058 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e422133551964617338a1f 2026-04-21 00:30:32,059 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:30:32,072 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e422133551964617338a1f HTTP/1.1" 200 None 2026-04-21 00:30:32,074 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:30:32,092 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 00:30:32,093 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:30:32,112 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e422133551964617338a1f HTTP/1.1" 200 None 2026-04-21 00:30:32,113 - lava_callback - ERROR - Error processing callback for node 69e422133551964617338a1f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:34172 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41926 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:30:52,091 - lava_callback - INFO - Received callback for node 69e422383551964617338a2f INFO: 172.20.0.1:34022 - "POST /node/69e422383551964617338a2f HTTP/1.0" 202 Accepted 2026-04-21 00:30:52,101 - lava_callback - INFO - Starting async job submit for node 69e422383551964617338a2f 2026-04-21 00:30:52,102 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:30:52,123 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e422383551964617338a2f HTTP/1.1" 200 None 2026-04-21 00:30:52,124 - lava_callback - DEBUG - Processing node 69e422383551964617338a2f: baseline-arm-clabbe 2026-04-21 00:30:52,125 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e422383551964617338a2f 2026-04-21 00:30:52,125 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:30:52,153 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e422383551964617338a2f HTTP/1.1" 200 None 2026-04-21 00:30:52,155 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:30:52,181 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 00:30:52,183 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:30:52,213 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e422383551964617338a2f HTTP/1.1" 200 None 2026-04-21 00:30:52,215 - lava_callback - ERROR - Error processing callback for node 69e422383551964617338a2f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:34024 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48950 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44818 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44834 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41124 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41136 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:31:32,109 - lava_callback - INFO - Received callback for node 69e4225d3551964617338a3c INFO: 172.20.0.1:48692 - "POST /node/69e4225d3551964617338a3c HTTP/1.0" 202 Accepted 2026-04-21 00:31:32,115 - lava_callback - INFO - Starting async job submit for node 69e4225d3551964617338a3c 2026-04-21 00:31:32,116 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:31:32,133 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4225d3551964617338a3c HTTP/1.1" 200 None 2026-04-21 00:31:32,134 - lava_callback - DEBUG - Processing node 69e4225d3551964617338a3c: baseline-arm-clabbe 2026-04-21 00:31:32,134 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4225d3551964617338a3c 2026-04-21 00:31:32,135 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:31:32,159 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4225d3551964617338a3c HTTP/1.1" 200 None 2026-04-21 00:31:32,162 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:31:32,180 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 00:31:32,182 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:31:32,203 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4225d3551964617338a3c HTTP/1.1" 200 None 2026-04-21 00:31:32,205 - lava_callback - ERROR - Error processing callback for node 69e4225d3551964617338a3c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:48700 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49608 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36126 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43304 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:32:12,150 - lava_callback - INFO - Received callback for node 69e422823551964617338a4e INFO: 172.20.0.1:49486 - "POST /node/69e422823551964617338a4e HTTP/1.0" 202 Accepted 2026-04-21 00:32:12,163 - lava_callback - INFO - Starting async job submit for node 69e422823551964617338a4e 2026-04-21 00:32:12,165 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:32:12,216 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e422823551964617338a4e HTTP/1.1" 200 None 2026-04-21 00:32:12,218 - lava_callback - DEBUG - Processing node 69e422823551964617338a4e: baseline-arm-clabbe 2026-04-21 00:32:12,218 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e422823551964617338a4e 2026-04-21 00:32:12,219 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:32:12,242 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e422823551964617338a4e HTTP/1.1" 200 None 2026-04-21 00:32:12,245 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:32:12,301 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 00:32:12,304 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:32:12,342 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e422823551964617338a4e HTTP/1.1" 200 None 2026-04-21 00:32:12,343 - lava_callback - ERROR - Error processing callback for node 69e422823551964617338a4e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49494 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49508 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57400 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57408 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:32:28,242 - lava_callback - INFO - Received callback for node 69e6c58e60f5826562c157d0 INFO: 172.20.0.1:57410 - "POST /node/69e6c58e60f5826562c157d0 HTTP/1.0" 202 Accepted 2026-04-21 00:32:28,248 - lava_callback - INFO - Starting async job submit for node 69e6c58e60f5826562c157d0 2026-04-21 00:32:28,251 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:32:28,271 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c58e60f5826562c157d0 HTTP/1.1" 200 None 2026-04-21 00:32:28,272 - lava_callback - DEBUG - Processing node 69e6c58e60f5826562c157d0: baseline-arm 2026-04-21 00:32:28,277 - lava_callback - WARNING - Failed to upload log 2026-04-21 00:32:28,280 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:32:28,399 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:32:28,400 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69e6c58e60f5826562c157d0/lava-logs.yaml 2026-04-21 00:32:28,402 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:32:28,510 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:32:28,512 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69e6c58e60f5826562c157d0/lava_callback.json.gz 2026-04-21 00:32:28,515 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:32:28,534 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c58e60f5826562c157d0 HTTP/1.1" 200 None 2026-04-21 00:32:28,535 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:32:28,549 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:32:28,551 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:32:28,570 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c58e60f5826562c157d0 HTTP/1.1" 200 None 2026-04-21 00:32:28,581 - lava_callback - INFO - Completed processing callback for node 69e6c58e60f5826562c157d0 INFO: 172.20.0.1:43844 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:32:44,166 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:32:44,181 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-04-21 00:32:44,311 - lava_callback - INFO - Received callback for node 69e422a23551964617338a5e INFO: 172.20.0.1:47788 - "POST /node/69e422a23551964617338a5e HTTP/1.0" 202 Accepted 2026-04-21 00:32:44,317 - lava_callback - INFO - Starting async job submit for node 69e422a23551964617338a5e 2026-04-21 00:32:44,319 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:32:44,344 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e422a23551964617338a5e HTTP/1.1" 200 None 2026-04-21 00:32:44,345 - lava_callback - DEBUG - Processing node 69e422a23551964617338a5e: baseline-arm-baylibre 2026-04-21 00:32:44,345 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e422a23551964617338a5e 2026-04-21 00:32:44,346 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:32:44,373 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e422a23551964617338a5e HTTP/1.1" 200 None 2026-04-21 00:32:44,376 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:32:44,406 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68a3f6535ffd5b3ed8af HTTP/1.1" 200 None 2026-04-21 00:32:44,408 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:32:44,435 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e422a23551964617338a5e HTTP/1.1" 200 None 2026-04-21 00:32:44,436 - lava_callback - ERROR - Error processing callback for node 69e422a23551964617338a5e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47800 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57130 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:32:56,782 - lava_callback - INFO - Received callback for node 69e6c59260f5826562c157dc INFO: 172.20.0.1:57136 - "POST /node/69e6c59260f5826562c157dc HTTP/1.0" 202 Accepted 2026-04-21 00:32:56,831 - lava_callback - INFO - Starting async job submit for node 69e6c59260f5826562c157dc 2026-04-21 00:32:56,861 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:32:56,890 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c59260f5826562c157dc HTTP/1.1" 200 None 2026-04-21 00:32:56,891 - lava_callback - DEBUG - Processing node 69e6c59260f5826562c157dc: baseline-arm 2026-04-21 00:32:57,102 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:32:57,203 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:32:57,205 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69e6c59260f5826562c157dc/log.txt.gz 2026-04-21 00:32:57,324 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:32:57,485 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:32:57,487 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69e6c59260f5826562c157dc/lava-logs.yaml 2026-04-21 00:32:57,492 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:32:57,602 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:32:57,605 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69e6c59260f5826562c157dc/lava_callback.json.gz 2026-04-21 00:32:57,631 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:32:57,653 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c59260f5826562c157dc HTTP/1.1" 200 None 2026-04-21 00:32:57,656 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:32:57,681 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:32:57,683 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:32:57,718 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c59260f5826562c157dc HTTP/1.1" 200 None 2026-04-21 00:32:57,719 - lava_callback - INFO - Completed processing callback for node 69e6c59260f5826562c157dc 2026-04-21 00:33:03,681 - lava_callback - INFO - Received callback for node 69e6c58f60f5826562c157d3 INFO: 172.20.0.1:59432 - "POST /node/69e6c58f60f5826562c157d3 HTTP/1.0" 202 Accepted 2026-04-21 00:33:03,731 - lava_callback - INFO - Starting async job submit for node 69e6c58f60f5826562c157d3 2026-04-21 00:33:03,752 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:03,793 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c58f60f5826562c157d3 HTTP/1.1" 200 None 2026-04-21 00:33:03,794 - lava_callback - DEBUG - Processing node 69e6c58f60f5826562c157d3: baseline-arm 2026-04-21 00:33:03,924 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:33:04,218 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:33:04,220 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69e6c58f60f5826562c157d3/log.txt.gz 2026-04-21 00:33:04,291 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:33:04,443 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:33:04,445 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69e6c58f60f5826562c157d3/lava-logs.yaml 2026-04-21 00:33:04,449 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:33:04,606 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:33:04,608 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69e6c58f60f5826562c157d3/lava_callback.json.gz 2026-04-21 00:33:04,644 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:04,667 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c58f60f5826562c157d3 HTTP/1.1" 200 None 2026-04-21 00:33:04,668 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:04,682 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:33:04,684 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:04,727 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c58f60f5826562c157d3 HTTP/1.1" 200 None 2026-04-21 00:33:04,728 - lava_callback - INFO - Completed processing callback for node 69e6c58f60f5826562c157d3 INFO: 172.20.0.1:59434 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:33:14,183 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:14,204 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:39172 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39186 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56840 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:33:24,342 - lava_callback - INFO - Received callback for node 69e422c73551964617338a6c INFO: 172.20.0.1:56854 - "POST /node/69e422c73551964617338a6c HTTP/1.0" 202 Accepted 2026-04-21 00:33:24,349 - lava_callback - INFO - Starting async job submit for node 69e422c73551964617338a6c 2026-04-21 00:33:24,350 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:24,369 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e422c73551964617338a6c HTTP/1.1" 200 None 2026-04-21 00:33:24,370 - lava_callback - DEBUG - Processing node 69e422c73551964617338a6c: baseline-arm-baylibre 2026-04-21 00:33:24,370 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e422c73551964617338a6c 2026-04-21 00:33:24,371 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:24,409 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e422c73551964617338a6c HTTP/1.1" 200 None 2026-04-21 00:33:24,412 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:24,433 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68a3f6535ffd5b3ed8af HTTP/1.1" 200 None 2026-04-21 00:33:24,435 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:24,460 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e422c73551964617338a6c HTTP/1.1" 200 None 2026-04-21 00:33:24,462 - lava_callback - ERROR - Error processing callback for node 69e422c73551964617338a6c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:56864 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:33:34,277 - lava_callback - INFO - Received callback for node 69e6c58e60f5826562c157cd INFO: 172.20.0.1:41378 - "POST /node/69e6c58e60f5826562c157cd HTTP/1.0" 202 Accepted 2026-04-21 00:33:34,288 - lava_callback - INFO - Starting async job submit for node 69e6c58e60f5826562c157cd 2026-04-21 00:33:34,323 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:34,344 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c58e60f5826562c157cd HTTP/1.1" 200 None 2026-04-21 00:33:34,345 - lava_callback - DEBUG - Processing node 69e6c58e60f5826562c157cd: baseline-arm 2026-04-21 00:33:34,559 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:33:34,647 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:33:34,649 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69e6c58e60f5826562c157cd/log.txt.gz 2026-04-21 00:33:34,768 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:33:34,897 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:33:34,899 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69e6c58e60f5826562c157cd/lava-logs.yaml 2026-04-21 00:33:34,904 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:33:34,986 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:33:34,988 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69e6c58e60f5826562c157cd/lava_callback.json.gz 2026-04-21 00:33:35,009 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:35,022 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c58e60f5826562c157cd HTTP/1.1" 200 None 2026-04-21 00:33:35,024 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:35,037 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:33:35,038 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:35,060 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c58e60f5826562c157cd HTTP/1.1" 200 None 2026-04-21 00:33:35,061 - lava_callback - INFO - Completed processing callback for node 69e6c58e60f5826562c157cd INFO: 172.20.0.1:41382 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:33:39,924 - lava_callback - INFO - Received callback for node 69e6c59160f5826562c157d9 INFO: 172.20.0.1:45156 - "POST /node/69e6c59160f5826562c157d9 HTTP/1.0" 202 Accepted 2026-04-21 00:33:39,930 - lava_callback - INFO - Starting async job submit for node 69e6c59160f5826562c157d9 2026-04-21 00:33:39,970 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:39,985 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c59160f5826562c157d9 HTTP/1.1" 200 None 2026-04-21 00:33:39,986 - lava_callback - DEBUG - Processing node 69e6c59160f5826562c157d9: baseline-arm 2026-04-21 00:33:40,202 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:33:40,283 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:33:40,284 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69e6c59160f5826562c157d9/log.txt.gz 2026-04-21 00:33:40,419 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:33:40,541 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:33:40,543 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69e6c59160f5826562c157d9/lava-logs.yaml 2026-04-21 00:33:40,549 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:33:40,637 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:33:40,639 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69e6c59160f5826562c157d9/lava_callback.json.gz 2026-04-21 00:33:40,661 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:40,675 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c59160f5826562c157d9 HTTP/1.1" 200 None 2026-04-21 00:33:40,676 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:40,693 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:33:40,695 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:40,717 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c59160f5826562c157d9 HTTP/1.1" 200 None 2026-04-21 00:33:40,718 - lava_callback - INFO - Completed processing callback for node 69e6c59160f5826562c157d9 2026-04-21 00:33:44,206 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:44,222 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-04-21 00:33:45,313 - lava_callback - INFO - Received callback for node 69e6c5b260f5826562c15871 INFO: 172.20.0.1:45164 - "POST /node/69e6c5b260f5826562c15871 HTTP/1.0" 202 Accepted 2026-04-21 00:33:45,361 - lava_callback - INFO - Starting async job submit for node 69e6c5b260f5826562c15871 2026-04-21 00:33:45,382 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:45,396 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5b260f5826562c15871 HTTP/1.1" 200 None 2026-04-21 00:33:45,397 - lava_callback - DEBUG - Processing node 69e6c5b260f5826562c15871: baseline-arm-pengutronix 2026-04-21 00:33:45,503 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:33:45,591 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:33:45,593 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69e6c5b260f5826562c15871/log.txt.gz 2026-04-21 00:33:45,654 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:33:45,783 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:33:45,785 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69e6c5b260f5826562c15871/lava-logs.yaml 2026-04-21 00:33:45,789 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:33:45,864 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:33:45,866 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69e6c5b260f5826562c15871/lava_callback.json.gz 2026-04-21 00:33:45,886 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:45,901 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5b260f5826562c15871 HTTP/1.1" 200 None 2026-04-21 00:33:45,902 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:45,917 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:33:45,918 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:45,939 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c5b260f5826562c15871 HTTP/1.1" 200 None 2026-04-21 00:33:45,940 - lava_callback - INFO - Completed processing callback for node 69e6c5b260f5826562c15871 INFO: 172.20.0.1:45172 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:33:47,624 - lava_callback - INFO - Received callback for node 69e6c5b060f5826562c1586b INFO: 172.20.0.1:45178 - "POST /node/69e6c5b060f5826562c1586b HTTP/1.0" 202 Accepted 2026-04-21 00:33:47,629 - lava_callback - INFO - Starting async job submit for node 69e6c5b060f5826562c1586b 2026-04-21 00:33:47,649 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:47,665 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5b060f5826562c1586b HTTP/1.1" 200 None 2026-04-21 00:33:47,665 - lava_callback - DEBUG - Processing node 69e6c5b060f5826562c1586b: baseline-arm-pengutronix 2026-04-21 00:33:47,824 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:33:47,907 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:33:47,909 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69e6c5b060f5826562c1586b/log.txt.gz 2026-04-21 00:33:47,991 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:33:48,112 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:33:48,114 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69e6c5b060f5826562c1586b/lava-logs.yaml 2026-04-21 00:33:48,119 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:33:48,208 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:33:48,210 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69e6c5b060f5826562c1586b/lava_callback.json.gz 2026-04-21 00:33:48,230 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:48,244 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5b060f5826562c1586b HTTP/1.1" 200 None 2026-04-21 00:33:48,245 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:48,258 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:33:48,259 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:48,280 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c5b060f5826562c1586b HTTP/1.1" 200 None 2026-04-21 00:33:48,281 - lava_callback - INFO - Completed processing callback for node 69e6c5b060f5826562c1586b 2026-04-21 00:33:50,970 - lava_callback - INFO - Received callback for node 69e6c5b160f5826562c1586e INFO: 172.20.0.1:39362 - "POST /node/69e6c5b160f5826562c1586e HTTP/1.0" 202 Accepted 2026-04-21 00:33:50,976 - lava_callback - INFO - Starting async job submit for node 69e6c5b160f5826562c1586e 2026-04-21 00:33:50,995 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:51,010 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5b160f5826562c1586e HTTP/1.1" 200 None 2026-04-21 00:33:51,011 - lava_callback - DEBUG - Processing node 69e6c5b160f5826562c1586e: baseline-arm-pengutronix 2026-04-21 00:33:51,164 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:33:51,237 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:33:51,238 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69e6c5b160f5826562c1586e/log.txt.gz 2026-04-21 00:33:51,342 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:33:51,555 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:33:51,557 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69e6c5b160f5826562c1586e/lava-logs.yaml 2026-04-21 00:33:51,562 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:33:51,651 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:33:51,653 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69e6c5b160f5826562c1586e/lava_callback.json.gz 2026-04-21 00:33:51,671 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:51,685 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5b160f5826562c1586e HTTP/1.1" 200 None 2026-04-21 00:33:51,686 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:51,701 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:33:51,702 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:33:51,730 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c5b160f5826562c1586e HTTP/1.1" 200 None 2026-04-21 00:33:51,731 - lava_callback - INFO - Completed processing callback for node 69e6c5b160f5826562c1586e INFO: 172.20.0.1:39372 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39162 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:34:12,172 - lava_callback - INFO - Received callback for node 69e422f03551964617338a79 INFO: 172.20.0.1:52168 - "POST /node/69e422f03551964617338a79 HTTP/1.0" 202 Accepted 2026-04-21 00:34:12,178 - lava_callback - INFO - Starting async job submit for node 69e422f03551964617338a79 2026-04-21 00:34:12,180 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:34:12,195 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e422f03551964617338a79 HTTP/1.1" 200 None 2026-04-21 00:34:12,196 - lava_callback - DEBUG - Processing node 69e422f03551964617338a79: baseline-arm-clabbe 2026-04-21 00:34:12,196 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e422f03551964617338a79 2026-04-21 00:34:12,197 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:34:12,220 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e422f03551964617338a79 HTTP/1.1" 200 None 2026-04-21 00:34:12,221 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:34:12,234 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 00:34:12,236 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:34:12,253 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e422f03551964617338a79 HTTP/1.1" 200 None 2026-04-21 00:34:12,254 - lava_callback - ERROR - Error processing callback for node 69e422f03551964617338a79: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-04-21 00:34:14,224 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:34:14,248 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:52178 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52186 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50298 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50300 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:34:28,099 - lava_callback - INFO - Received callback for node 69e6c5ac60f5826562c1585d INFO: 172.20.0.1:50304 - "POST /node/69e6c5ac60f5826562c1585d HTTP/1.0" 202 Accepted 2026-04-21 00:34:28,107 - lava_callback - INFO - Starting async job submit for node 69e6c5ac60f5826562c1585d 2026-04-21 00:34:28,134 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:34:28,150 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5ac60f5826562c1585d HTTP/1.1" 200 None 2026-04-21 00:34:28,151 - lava_callback - DEBUG - Processing node 69e6c5ac60f5826562c1585d: baseline-arm-kontron 2026-04-21 00:34:28,355 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:34:28,446 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:34:28,448 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-kontron-69e6c5ac60f5826562c1585d/log.txt.gz 2026-04-21 00:34:28,552 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:34:28,712 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:34:28,714 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-kontron-69e6c5ac60f5826562c1585d/lava-logs.yaml 2026-04-21 00:34:28,719 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:34:28,806 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:34:28,808 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-kontron-69e6c5ac60f5826562c1585d/lava_callback.json.gz 2026-04-21 00:34:28,840 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:34:28,864 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5ac60f5826562c1585d HTTP/1.1" 200 None 2026-04-21 00:34:28,867 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:34:28,884 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:34:28,886 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:34:28,914 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c5ac60f5826562c1585d HTTP/1.1" 200 None 2026-04-21 00:34:28,917 - lava_callback - INFO - Completed processing callback for node 69e6c5ac60f5826562c1585d 2026-04-21 00:34:29,393 - lava_callback - INFO - Received callback for node 69e6c5ad60f5826562c15860 INFO: 172.20.0.1:50314 - "POST /node/69e6c5ad60f5826562c15860 HTTP/1.0" 202 Accepted 2026-04-21 00:34:29,400 - lava_callback - INFO - Starting async job submit for node 69e6c5ad60f5826562c15860 2026-04-21 00:34:29,436 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:34:29,451 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5ad60f5826562c15860 HTTP/1.1" 200 None 2026-04-21 00:34:29,452 - lava_callback - DEBUG - Processing node 69e6c5ad60f5826562c15860: baseline-arm-kontron 2026-04-21 00:34:29,660 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:34:29,766 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:34:29,768 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-kontron-69e6c5ad60f5826562c15860/log.txt.gz 2026-04-21 00:34:29,897 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:34:30,047 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:34:30,050 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-kontron-69e6c5ad60f5826562c15860/lava-logs.yaml 2026-04-21 00:34:30,055 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:34:30,168 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:34:30,169 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-kontron-69e6c5ad60f5826562c15860/lava_callback.json.gz 2026-04-21 00:34:30,192 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:34:30,206 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5ad60f5826562c15860 HTTP/1.1" 200 None 2026-04-21 00:34:30,208 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:34:30,222 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:34:30,223 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:34:30,257 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c5ad60f5826562c15860 HTTP/1.1" 200 None 2026-04-21 00:34:30,259 - lava_callback - INFO - Completed processing callback for node 69e6c5ad60f5826562c15860 INFO: 172.20.0.1:37568 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:34:44,250 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:34:44,266 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:38230 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:34:52,161 - lava_callback - INFO - Received callback for node 69e423153551964617338a88 INFO: 172.20.0.1:54100 - "POST /node/69e423153551964617338a88 HTTP/1.0" 202 Accepted 2026-04-21 00:34:52,167 - lava_callback - INFO - Starting async job submit for node 69e423153551964617338a88 2026-04-21 00:34:52,168 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:34:52,181 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e423153551964617338a88 HTTP/1.1" 200 None 2026-04-21 00:34:52,182 - lava_callback - DEBUG - Processing node 69e423153551964617338a88: baseline-arm-clabbe 2026-04-21 00:34:52,182 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e423153551964617338a88 2026-04-21 00:34:52,183 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:34:52,196 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e423153551964617338a88 HTTP/1.1" 200 None 2026-04-21 00:34:52,197 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:34:52,210 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 00:34:52,211 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:34:52,230 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e423153551964617338a88 HTTP/1.1" 200 None 2026-04-21 00:34:52,231 - lava_callback - ERROR - Error processing callback for node 69e423153551964617338a88: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54102 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:35:04,436 - lava_callback - INFO - Received callback for node 69e423363551964617338a95 INFO: 172.20.0.1:35032 - "POST /node/69e423363551964617338a95 HTTP/1.0" 202 Accepted 2026-04-21 00:35:04,442 - lava_callback - INFO - Starting async job submit for node 69e423363551964617338a95 2026-04-21 00:35:04,443 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:35:04,456 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e423363551964617338a95 HTTP/1.1" 200 None 2026-04-21 00:35:04,457 - lava_callback - DEBUG - Processing node 69e423363551964617338a95: baseline-arm-baylibre 2026-04-21 00:35:04,457 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e423363551964617338a95 2026-04-21 00:35:04,458 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:35:04,471 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e423363551964617338a95 HTTP/1.1" 200 None 2026-04-21 00:35:04,472 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:35:04,486 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68a3f6535ffd5b3ed8af HTTP/1.1" 200 None 2026-04-21 00:35:04,487 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:35:04,508 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e423363551964617338a95 HTTP/1.1" 200 None 2026-04-21 00:35:04,509 - lava_callback - ERROR - Error processing callback for node 69e423363551964617338a95: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:35042 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:35:14,658 - lava_callback - INFO - Received callback for node 69e6c5cb60f5826562c158b3 INFO: 172.20.0.1:47516 - "POST /node/69e6c5cb60f5826562c158b3 HTTP/1.0" 202 Accepted 2026-04-21 00:35:14,665 - lava_callback - INFO - Starting async job submit for node 69e6c5cb60f5826562c158b3 2026-04-21 00:35:14,688 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:35:14,703 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5cb60f5826562c158b3 HTTP/1.1" 200 None 2026-04-21 00:35:14,704 - lava_callback - DEBUG - Processing node 69e6c5cb60f5826562c158b3: ltp-dio 2026-04-21 00:35:15,086 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.20.0.1:47520 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 00:35:15,258 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:35:15,259 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-dio-69e6c5cb60f5826562c158b3/log.txt.gz 2026-04-21 00:35:15,485 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:35:15,661 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:35:15,663 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-dio-69e6c5cb60f5826562c158b3/lava-logs.yaml 2026-04-21 00:35:15,673 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:35:15,782 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:35:15,784 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-dio-69e6c5cb60f5826562c158b3/lava_callback.json.gz 2026-04-21 00:35:15,807 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:35:15,821 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5cb60f5826562c158b3 HTTP/1.1" 200 None 2026-04-21 00:35:15,823 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:35:15,836 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:35:15,837 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:35:15,858 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c5cb60f5826562c158b3 HTTP/1.1" 200 None 2026-04-21 00:35:15,859 - lava_callback - INFO - Completed processing callback for node 69e6c5cb60f5826562c158b3 INFO: 172.20.0.1:47532 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59052 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59060 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48316 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:35:44,268 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:35:44,285 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-04-21 00:35:44,451 - lava_callback - INFO - Received callback for node 69e4235b3551964617338aa4 INFO: 172.20.0.1:45870 - "POST /node/69e4235b3551964617338aa4 HTTP/1.0" 202 Accepted 2026-04-21 00:35:44,457 - lava_callback - INFO - Starting async job submit for node 69e4235b3551964617338aa4 2026-04-21 00:35:44,458 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:35:44,472 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4235b3551964617338aa4 HTTP/1.1" 200 None 2026-04-21 00:35:44,473 - lava_callback - DEBUG - Processing node 69e4235b3551964617338aa4: baseline-arm-baylibre 2026-04-21 00:35:44,473 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4235b3551964617338aa4 2026-04-21 00:35:44,474 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:35:44,514 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4235b3551964617338aa4 HTTP/1.1" 200 None 2026-04-21 00:35:44,524 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:35:44,546 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68a3f6535ffd5b3ed8af HTTP/1.1" 200 None 2026-04-21 00:35:44,548 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:35:44,578 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4235b3551964617338aa4 HTTP/1.1" 200 None 2026-04-21 00:35:44,579 - lava_callback - ERROR - Error processing callback for node 69e4235b3551964617338aa4: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:45878 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50836 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60066 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35192 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35204 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39474 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:36:24,510 - lava_callback - INFO - Received callback for node 69e423803551964617338ab0 INFO: 172.20.0.1:39484 - "POST /node/69e423803551964617338ab0 HTTP/1.0" 202 Accepted 2026-04-21 00:36:24,515 - lava_callback - INFO - Starting async job submit for node 69e423803551964617338ab0 2026-04-21 00:36:24,516 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:36:24,531 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e423803551964617338ab0 HTTP/1.1" 200 None 2026-04-21 00:36:24,532 - lava_callback - DEBUG - Processing node 69e423803551964617338ab0: baseline-arm-baylibre 2026-04-21 00:36:24,532 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e423803551964617338ab0 2026-04-21 00:36:24,533 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:36:24,547 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e423803551964617338ab0 HTTP/1.1" 200 None 2026-04-21 00:36:24,548 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:36:24,561 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68a3f6535ffd5b3ed8af HTTP/1.1" 200 None 2026-04-21 00:36:24,562 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:36:24,580 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e423803551964617338ab0 HTTP/1.1" 200 None 2026-04-21 00:36:24,581 - lava_callback - ERROR - Error processing callback for node 69e423803551964617338ab0: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:39496 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38422 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:36:41,468 - lava_callback - INFO - Received callback for node 69e6c5a160f5826562c15804 INFO: 172.20.0.1:36494 - "POST /node/69e6c5a160f5826562c15804 HTTP/1.0" 202 Accepted 2026-04-21 00:36:41,475 - lava_callback - INFO - Starting async job submit for node 69e6c5a160f5826562c15804 2026-04-21 00:36:41,498 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:36:41,512 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5a160f5826562c15804 HTTP/1.1" 200 None 2026-04-21 00:36:41,513 - lava_callback - DEBUG - Processing node 69e6c5a160f5826562c15804: baseline-arm-broonie 2026-04-21 00:36:41,744 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:36:41,967 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:36:41,969 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69e6c5a160f5826562c15804/log.txt.gz 2026-04-21 00:36:42,094 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:36:42,270 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:36:42,272 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69e6c5a160f5826562c15804/lava-logs.yaml 2026-04-21 00:36:42,278 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:36:42,494 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:36:42,495 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69e6c5a160f5826562c15804/lava_callback.json.gz 2026-04-21 00:36:42,518 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:36:42,532 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5a160f5826562c15804 HTTP/1.1" 200 None 2026-04-21 00:36:42,534 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:36:42,547 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:36:42,549 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:36:42,570 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c5a160f5826562c15804 HTTP/1.1" 200 None 2026-04-21 00:36:42,571 - lava_callback - INFO - Completed processing callback for node 69e6c5a160f5826562c15804 2026-04-21 00:36:44,287 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:36:44,306 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:36508 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55252 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:37:04,492 - lava_callback - INFO - Received callback for node 69e423a53551964617338abf INFO: 172.20.0.1:44754 - "POST /node/69e423a53551964617338abf HTTP/1.0" 202 Accepted 2026-04-21 00:37:04,498 - lava_callback - INFO - Starting async job submit for node 69e423a53551964617338abf 2026-04-21 00:37:04,499 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:37:04,513 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e423a53551964617338abf HTTP/1.1" 200 None 2026-04-21 00:37:04,513 - lava_callback - DEBUG - Processing node 69e423a53551964617338abf: baseline-arm-baylibre 2026-04-21 00:37:04,514 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e423a53551964617338abf 2026-04-21 00:37:04,514 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:37:04,530 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e423a53551964617338abf HTTP/1.1" 200 None 2026-04-21 00:37:04,532 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:37:04,544 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68a3f6535ffd5b3ed8af HTTP/1.1" 200 None 2026-04-21 00:37:04,546 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:37:04,562 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e423a53551964617338abf HTTP/1.1" 200 None 2026-04-21 00:37:04,564 - lava_callback - ERROR - Error processing callback for node 69e423a53551964617338abf: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:44760 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51944 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51954 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52760 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:37:21,796 - lava_callback - INFO - Received callback for node 69e6c5a060f5826562c15801 INFO: 172.20.0.1:52772 - "POST /node/69e6c5a060f5826562c15801 HTTP/1.0" 202 Accepted 2026-04-21 00:37:21,802 - lava_callback - INFO - Starting async job submit for node 69e6c5a060f5826562c15801 2026-04-21 00:37:21,826 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:37:21,840 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5a060f5826562c15801 HTTP/1.1" 200 None 2026-04-21 00:37:21,841 - lava_callback - DEBUG - Processing node 69e6c5a060f5826562c15801: baseline-arm-broonie 2026-04-21 00:37:22,046 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:37:22,161 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:37:22,162 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69e6c5a060f5826562c15801/log.txt.gz 2026-04-21 00:37:22,282 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:37:22,611 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:37:22,613 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69e6c5a060f5826562c15801/lava-logs.yaml 2026-04-21 00:37:22,619 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:37:22,725 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:37:22,727 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69e6c5a060f5826562c15801/lava_callback.json.gz 2026-04-21 00:37:22,750 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:37:22,763 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5a060f5826562c15801 HTTP/1.1" 200 None 2026-04-21 00:37:22,765 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:37:22,778 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:37:22,779 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:37:22,809 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c5a060f5826562c15801 HTTP/1.1" 200 None 2026-04-21 00:37:22,811 - lava_callback - INFO - Completed processing callback for node 69e6c5a060f5826562c15801 INFO: 172.20.0.1:52776 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60374 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:37:44,308 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:37:44,324 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-04-21 00:37:44,659 - lava_callback - INFO - Received callback for node 69e423c63551964617338acc INFO: 172.20.0.1:48012 - "POST /node/69e423c63551964617338acc HTTP/1.0" 202 Accepted 2026-04-21 00:37:44,669 - lava_callback - INFO - Starting async job submit for node 69e423c63551964617338acc 2026-04-21 00:37:44,670 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:37:44,694 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e423c63551964617338acc HTTP/1.1" 200 None 2026-04-21 00:37:44,695 - lava_callback - DEBUG - Processing node 69e423c63551964617338acc: baseline-arm-baylibre 2026-04-21 00:37:44,695 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e423c63551964617338acc 2026-04-21 00:37:44,696 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:37:44,713 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e423c63551964617338acc HTTP/1.1" 200 None 2026-04-21 00:37:44,715 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:37:44,745 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68a3f6535ffd5b3ed8af HTTP/1.1" 200 None 2026-04-21 00:37:44,748 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:37:44,771 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e423c63551964617338acc HTTP/1.1" 200 None 2026-04-21 00:37:44,772 - lava_callback - ERROR - Error processing callback for node 69e423c63551964617338acc: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:48026 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60612 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42476 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:38:12,400 - lava_callback - INFO - Received callback for node 69e423ef3551964617338ade INFO: 172.20.0.1:46472 - "POST /node/69e423ef3551964617338ade HTTP/1.0" 202 Accepted 2026-04-21 00:38:12,410 - lava_callback - INFO - Starting async job submit for node 69e423ef3551964617338ade 2026-04-21 00:38:12,411 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:38:12,425 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e423ef3551964617338ade HTTP/1.1" 200 None 2026-04-21 00:38:12,426 - lava_callback - DEBUG - Processing node 69e423ef3551964617338ade: baseline-arm-clabbe 2026-04-21 00:38:12,426 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e423ef3551964617338ade 2026-04-21 00:38:12,427 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:38:12,439 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e423ef3551964617338ade HTTP/1.1" 200 None 2026-04-21 00:38:12,441 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:38:12,454 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68a3f6535ffd5b3ed8af HTTP/1.1" 200 None 2026-04-21 00:38:12,456 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:38:12,479 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e423ef3551964617338ade HTTP/1.1" 200 None 2026-04-21 00:38:12,481 - lava_callback - ERROR - Error processing callback for node 69e423ef3551964617338ade: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:46486 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46498 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49104 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49110 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56922 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:38:39,017 - lava_callback - INFO - Received callback for node 69e6c5a360f5826562c15807 INFO: 172.20.0.1:56924 - "POST /node/69e6c5a360f5826562c15807 HTTP/1.0" 202 Accepted 2026-04-21 00:38:39,023 - lava_callback - INFO - Starting async job submit for node 69e6c5a360f5826562c15807 2026-04-21 00:38:39,046 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:38:39,066 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5a360f5826562c15807 HTTP/1.1" 200 None 2026-04-21 00:38:39,066 - lava_callback - DEBUG - Processing node 69e6c5a360f5826562c15807: baseline-arm-broonie 2026-04-21 00:38:39,315 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:38:39,434 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:38:39,436 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69e6c5a360f5826562c15807/log.txt.gz 2026-04-21 00:38:39,616 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:38:39,778 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:38:39,779 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69e6c5a360f5826562c15807/lava-logs.yaml 2026-04-21 00:38:39,786 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:38:40,068 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:38:40,070 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69e6c5a360f5826562c15807/lava_callback.json.gz 2026-04-21 00:38:40,093 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:38:40,106 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5a360f5826562c15807 HTTP/1.1" 200 None 2026-04-21 00:38:40,108 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:38:40,122 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:38:40,124 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:38:40,145 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c5a360f5826562c15807 HTTP/1.1" 200 None 2026-04-21 00:38:40,147 - lava_callback - INFO - Completed processing callback for node 69e6c5a360f5826562c15807 2026-04-21 00:38:44,326 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:38:44,342 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:45118 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:38:52,382 - lava_callback - INFO - Received callback for node 69e424143551964617338aea INFO: 172.20.0.1:43926 - "POST /node/69e424143551964617338aea HTTP/1.0" 202 Accepted 2026-04-21 00:38:52,387 - lava_callback - INFO - Starting async job submit for node 69e424143551964617338aea 2026-04-21 00:38:52,388 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:38:52,402 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e424143551964617338aea HTTP/1.1" 200 None 2026-04-21 00:38:52,403 - lava_callback - DEBUG - Processing node 69e424143551964617338aea: baseline-arm-clabbe 2026-04-21 00:38:52,403 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e424143551964617338aea 2026-04-21 00:38:52,404 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:38:52,417 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e424143551964617338aea HTTP/1.1" 200 None 2026-04-21 00:38:52,419 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:38:52,432 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68a3f6535ffd5b3ed8af HTTP/1.1" 200 None 2026-04-21 00:38:52,434 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:38:52,451 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e424143551964617338aea HTTP/1.1" 200 None 2026-04-21 00:38:52,452 - lava_callback - ERROR - Error processing callback for node 69e424143551964617338aea: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:43928 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41598 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50356 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50358 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55412 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55416 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:39:32,421 - lava_callback - INFO - Received callback for node 69e424383551964617338af7 INFO: 172.20.0.1:49384 - "POST /node/69e424383551964617338af7 HTTP/1.0" 202 Accepted 2026-04-21 00:39:32,427 - lava_callback - INFO - Starting async job submit for node 69e424383551964617338af7 2026-04-21 00:39:32,428 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:39:32,441 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e424383551964617338af7 HTTP/1.1" 200 None 2026-04-21 00:39:32,442 - lava_callback - DEBUG - Processing node 69e424383551964617338af7: baseline-arm-clabbe 2026-04-21 00:39:32,442 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e424383551964617338af7 2026-04-21 00:39:32,443 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:39:32,459 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e424383551964617338af7 HTTP/1.1" 200 None 2026-04-21 00:39:32,460 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:39:32,473 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68a3f6535ffd5b3ed8af HTTP/1.1" 200 None 2026-04-21 00:39:32,474 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:39:32,492 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e424383551964617338af7 HTTP/1.1" 200 None 2026-04-21 00:39:32,493 - lava_callback - ERROR - Error processing callback for node 69e424383551964617338af7: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49388 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57242 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37332 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54668 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:40:12,455 - lava_callback - INFO - Received callback for node 69e4245d3551964617338b06 INFO: 172.20.0.1:57636 - "POST /node/69e4245d3551964617338b06 HTTP/1.0" 202 Accepted 2026-04-21 00:40:12,461 - lava_callback - INFO - Starting async job submit for node 69e4245d3551964617338b06 2026-04-21 00:40:12,462 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:40:12,480 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4245d3551964617338b06 HTTP/1.1" 200 None 2026-04-21 00:40:12,481 - lava_callback - DEBUG - Processing node 69e4245d3551964617338b06: baseline-arm-clabbe 2026-04-21 00:40:12,481 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4245d3551964617338b06 2026-04-21 00:40:12,482 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:40:12,495 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4245d3551964617338b06 HTTP/1.1" 200 None 2026-04-21 00:40:12,496 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:40:12,509 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68a3f6535ffd5b3ed8af HTTP/1.1" 200 None 2026-04-21 00:40:12,510 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:40:12,527 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4245d3551964617338b06 HTTP/1.1" 200 None 2026-04-21 00:40:12,528 - lava_callback - ERROR - Error processing callback for node 69e4245d3551964617338b06: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:57644 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57646 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42428 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42434 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36098 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42878 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:40:52,514 - lava_callback - INFO - Received callback for node 69e424813551964617338b16 INFO: 172.20.0.1:34650 - "POST /node/69e424813551964617338b16 HTTP/1.0" 202 Accepted 2026-04-21 00:40:52,519 - lava_callback - INFO - Starting async job submit for node 69e424813551964617338b16 2026-04-21 00:40:52,520 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:40:52,546 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e424813551964617338b16 HTTP/1.1" 200 None 2026-04-21 00:40:52,547 - lava_callback - DEBUG - Processing node 69e424813551964617338b16: baseline-arm-clabbe 2026-04-21 00:40:52,548 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e424813551964617338b16 2026-04-21 00:40:52,549 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:40:52,563 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e424813551964617338b16 HTTP/1.1" 200 None 2026-04-21 00:40:52,564 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:40:52,577 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68a3f6535ffd5b3ed8af HTTP/1.1" 200 None 2026-04-21 00:40:52,578 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:40:52,595 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e424813551964617338b16 HTTP/1.1" 200 None 2026-04-21 00:40:52,596 - lava_callback - ERROR - Error processing callback for node 69e424813551964617338b16: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:34662 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60012 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55958 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:55966 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51390 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51396 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:41:32,518 - lava_callback - INFO - Received callback for node 69e424a63551964617338b22 INFO: 172.20.0.1:39236 - "POST /node/69e424a63551964617338b22 HTTP/1.0" 202 Accepted 2026-04-21 00:41:32,527 - lava_callback - INFO - Starting async job submit for node 69e424a63551964617338b22 2026-04-21 00:41:32,528 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:41:32,541 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e424a63551964617338b22 HTTP/1.1" 200 None 2026-04-21 00:41:32,542 - lava_callback - DEBUG - Processing node 69e424a63551964617338b22: baseline-arm-clabbe 2026-04-21 00:41:32,542 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e424a63551964617338b22 2026-04-21 00:41:32,543 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:41:32,555 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e424a63551964617338b22 HTTP/1.1" 200 None 2026-04-21 00:41:32,557 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:41:32,576 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68a3f6535ffd5b3ed8af HTTP/1.1" 200 None 2026-04-21 00:41:32,578 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:41:32,595 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e424a63551964617338b22 HTTP/1.1" 200 None 2026-04-21 00:41:32,596 - lava_callback - ERROR - Error processing callback for node 69e424a63551964617338b22: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:39238 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:41:44,867 - lava_callback - INFO - Received callback for node 69e424c63551964617338b2e INFO: 172.20.0.1:44266 - "POST /node/69e424c63551964617338b2e HTTP/1.0" 202 Accepted 2026-04-21 00:41:44,873 - lava_callback - INFO - Starting async job submit for node 69e424c63551964617338b2e 2026-04-21 00:41:44,874 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:41:44,889 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e424c63551964617338b2e HTTP/1.1" 200 None 2026-04-21 00:41:44,890 - lava_callback - DEBUG - Processing node 69e424c63551964617338b2e: baseline-arm-baylibre 2026-04-21 00:41:44,890 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e424c63551964617338b2e 2026-04-21 00:41:44,890 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:41:44,905 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e424c63551964617338b2e HTTP/1.1" 200 None 2026-04-21 00:41:44,907 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:41:44,959 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68a3f6535ffd5b3ed8af HTTP/1.1" 200 None 2026-04-21 00:41:44,963 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:41:44,990 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e424c63551964617338b2e HTTP/1.1" 200 None 2026-04-21 00:41:44,995 - lava_callback - ERROR - Error processing callback for node 69e424c63551964617338b2e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:44278 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37628 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:41:59,802 - lava_callback - INFO - Received callback for node 69e6c5d760f5826562c158cc INFO: 172.20.0.1:51956 - "POST /node/69e6c5d760f5826562c158cc HTTP/1.0" 202 Accepted 2026-04-21 00:41:59,809 - lava_callback - INFO - Starting async job submit for node 69e6c5d760f5826562c158cc 2026-04-21 00:41:59,834 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:41:59,851 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5d760f5826562c158cc HTTP/1.1" 200 None 2026-04-21 00:41:59,852 - lava_callback - DEBUG - Processing node 69e6c5d760f5826562c158cc: ltp-syscalls 2026-04-21 00:42:00,560 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:42:00,704 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:42:00,706 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-syscalls-69e6c5d760f5826562c158cc/log.txt.gz 2026-04-21 00:42:01,185 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:42:01,389 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:42:01,391 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-syscalls-69e6c5d760f5826562c158cc/lava-logs.yaml 2026-04-21 00:42:01,409 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:42:01,523 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:42:01,525 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-syscalls-69e6c5d760f5826562c158cc/lava_callback.json.gz 2026-04-21 00:42:01,550 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:42:01,564 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5d760f5826562c158cc HTTP/1.1" 200 None 2026-04-21 00:42:01,565 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:42:01,578 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:42:01,580 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:42:01,599 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c5d760f5826562c158cc HTTP/1.1" 200 None 2026-04-21 00:42:01,601 - lava_callback - INFO - Completed processing callback for node 69e6c5d760f5826562c158cc INFO: 172.20.0.1:51966 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:42:14,345 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:42:14,360 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:46996 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47010 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53782 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53792 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:42:32,511 - lava_callback - INFO - Received callback for node 69e424ef3551964617338b3b INFO: 172.20.0.1:50964 - "POST /node/69e424ef3551964617338b3b HTTP/1.0" 202 Accepted 2026-04-21 00:42:32,517 - lava_callback - INFO - Starting async job submit for node 69e424ef3551964617338b3b 2026-04-21 00:42:32,518 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:42:32,532 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e424ef3551964617338b3b HTTP/1.1" 200 None 2026-04-21 00:42:32,533 - lava_callback - DEBUG - Processing node 69e424ef3551964617338b3b: baseline-arm-clabbe 2026-04-21 00:42:32,533 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e424ef3551964617338b3b 2026-04-21 00:42:32,533 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:42:32,546 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e424ef3551964617338b3b HTTP/1.1" 200 None 2026-04-21 00:42:32,547 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:42:32,559 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68a3f6535ffd5b3ed8af HTTP/1.1" 200 None 2026-04-21 00:42:32,561 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:42:32,578 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e424ef3551964617338b3b HTTP/1.1" 200 None 2026-04-21 00:42:32,579 - lava_callback - ERROR - Error processing callback for node 69e424ef3551964617338b3b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:50974 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45360 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51884 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58362 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:43:12,529 - lava_callback - INFO - Received callback for node 69e425143551964617338b48 INFO: 172.20.0.1:40622 - "POST /node/69e425143551964617338b48 HTTP/1.0" 202 Accepted 2026-04-21 00:43:12,537 - lava_callback - INFO - Starting async job submit for node 69e425143551964617338b48 2026-04-21 00:43:12,538 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:43:12,552 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e425143551964617338b48 HTTP/1.1" 200 None 2026-04-21 00:43:12,553 - lava_callback - DEBUG - Processing node 69e425143551964617338b48: baseline-arm-clabbe 2026-04-21 00:43:12,553 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e425143551964617338b48 2026-04-21 00:43:12,554 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:43:12,567 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e425143551964617338b48 HTTP/1.1" 200 None 2026-04-21 00:43:12,569 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:43:12,590 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68a3f6535ffd5b3ed8af HTTP/1.1" 200 None 2026-04-21 00:43:12,593 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:43:12,610 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e425143551964617338b48 HTTP/1.1" 200 None 2026-04-21 00:43:12,612 - lava_callback - ERROR - Error processing callback for node 69e425143551964617338b48: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:40624 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40634 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46898 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46912 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60158 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50540 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:43:52,611 - lava_callback - INFO - Received callback for node 69e425383551964617338b57 INFO: 172.20.0.1:34336 - "POST /node/69e425383551964617338b57 HTTP/1.0" 202 Accepted 2026-04-21 00:43:52,621 - lava_callback - INFO - Starting async job submit for node 69e425383551964617338b57 2026-04-21 00:43:52,622 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:43:52,644 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e425383551964617338b57 HTTP/1.1" 200 None 2026-04-21 00:43:52,645 - lava_callback - DEBUG - Processing node 69e425383551964617338b57: baseline-arm-clabbe 2026-04-21 00:43:52,646 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e425383551964617338b57 2026-04-21 00:43:52,646 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:43:52,665 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e425383551964617338b57 HTTP/1.1" 200 None 2026-04-21 00:43:52,667 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:43:52,686 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68a3f6535ffd5b3ed8af HTTP/1.1" 200 None 2026-04-21 00:43:52,688 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:43:52,734 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e425383551964617338b57 HTTP/1.1" 200 None 2026-04-21 00:43:52,736 - lava_callback - ERROR - Error processing callback for node 69e425383551964617338b57: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:34338 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57964 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59120 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59124 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:44:19,447 - lava_callback - INFO - Received callback for node 69e6c59e60f5826562c157fe INFO: 172.20.0.1:59130 - "POST /node/69e6c59e60f5826562c157fe HTTP/1.0" 202 Accepted 2026-04-21 00:44:19,454 - lava_callback - INFO - Starting async job submit for node 69e6c59e60f5826562c157fe 2026-04-21 00:44:19,495 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:44:19,518 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c59e60f5826562c157fe HTTP/1.1" 200 None 2026-04-21 00:44:19,519 - lava_callback - DEBUG - Processing node 69e6c59e60f5826562c157fe: baseline-arm-broonie 2026-04-21 00:44:19,734 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:44:19,835 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:44:19,837 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69e6c59e60f5826562c157fe/log.txt.gz 2026-04-21 00:44:19,965 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:44:20,121 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:44:20,123 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69e6c59e60f5826562c157fe/lava-logs.yaml 2026-04-21 00:44:20,128 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.20.0.1:54786 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:44:20,241 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:44:20,243 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69e6c59e60f5826562c157fe/lava_callback.json.gz 2026-04-21 00:44:20,266 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:44:20,279 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c59e60f5826562c157fe HTTP/1.1" 200 None 2026-04-21 00:44:20,281 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:44:20,321 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:44:20,323 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:44:20,344 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c59e60f5826562c157fe HTTP/1.1" 200 None 2026-04-21 00:44:20,346 - lava_callback - INFO - Completed processing callback for node 69e6c59e60f5826562c157fe 2026-04-21 00:44:24,993 - lava_callback - INFO - Received callback for node 69e425573551964617338b62 INFO: 172.20.0.1:54792 - "POST /node/69e425573551964617338b62 HTTP/1.0" 202 Accepted 2026-04-21 00:44:24,998 - lava_callback - INFO - Starting async job submit for node 69e425573551964617338b62 2026-04-21 00:44:24,999 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:44:25,015 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e425573551964617338b62 HTTP/1.1" 200 None 2026-04-21 00:44:25,016 - lava_callback - DEBUG - Processing node 69e425573551964617338b62: baseline-arm-baylibre 2026-04-21 00:44:25,016 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e425573551964617338b62 2026-04-21 00:44:25,016 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:44:25,033 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e425573551964617338b62 HTTP/1.1" 200 None 2026-04-21 00:44:25,035 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:44:25,055 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 00:44:25,057 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:44:25,078 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e425573551964617338b62 HTTP/1.1" 200 None 2026-04-21 00:44:25,079 - lava_callback - ERROR - Error processing callback for node 69e425573551964617338b62: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54806 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35692 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:44:44,362 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:44:44,379 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:51088 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:44:48,395 - lava_callback - INFO - Received callback for node 69e6c87360f5826562c15a50 INFO: 172.20.0.1:51092 - "POST /node/69e6c87360f5826562c15a50 HTTP/1.0" 202 Accepted 2026-04-21 00:44:48,401 - lava_callback - INFO - Starting async job submit for node 69e6c87360f5826562c15a50 2026-04-21 00:44:48,404 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:44:48,418 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c87360f5826562c15a50 HTTP/1.1" 200 None 2026-04-21 00:44:48,419 - lava_callback - DEBUG - Processing node 69e6c87360f5826562c15a50: baseline-arm 2026-04-21 00:44:48,425 - lava_callback - WARNING - Failed to upload log 2026-04-21 00:44:48,429 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:44:48,543 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:44:48,545 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69e6c87360f5826562c15a50/lava-logs.yaml 2026-04-21 00:44:48,546 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:44:48,663 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:44:48,664 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69e6c87360f5826562c15a50/lava_callback.json.gz 2026-04-21 00:44:48,667 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:44:48,681 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c87360f5826562c15a50 HTTP/1.1" 200 None 2026-04-21 00:44:48,682 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:44:48,695 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:44:48,697 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:44:48,719 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c87360f5826562c15a50 HTTP/1.1" 200 None 2026-04-21 00:44:48,729 - lava_callback - INFO - Completed processing callback for node 69e6c87360f5826562c15a50 INFO: 172.20.0.1:43388 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:45:04,941 - lava_callback - INFO - Received callback for node 69e4257c3551964617338b71 INFO: 172.20.0.1:50206 - "POST /node/69e4257c3551964617338b71 HTTP/1.0" 202 Accepted 2026-04-21 00:45:04,946 - lava_callback - INFO - Starting async job submit for node 69e4257c3551964617338b71 2026-04-21 00:45:04,947 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:45:04,976 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4257c3551964617338b71 HTTP/1.1" 200 None 2026-04-21 00:45:04,977 - lava_callback - DEBUG - Processing node 69e4257c3551964617338b71: baseline-arm-baylibre 2026-04-21 00:45:04,977 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4257c3551964617338b71 2026-04-21 00:45:04,978 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:45:04,991 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4257c3551964617338b71 HTTP/1.1" 200 None 2026-04-21 00:45:04,993 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:45:05,006 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 00:45:05,008 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:45:05,030 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4257c3551964617338b71 HTTP/1.1" 200 None 2026-04-21 00:45:05,031 - lava_callback - ERROR - Error processing callback for node 69e4257c3551964617338b71: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:50210 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:45:14,381 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:45:14,397 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:50714 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50716 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37510 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:45:24,909 - lava_callback - INFO - Received callback for node 69e425a03551964617338b7f INFO: 172.20.0.1:37512 - "POST /node/69e425a03551964617338b7f HTTP/1.0" 202 Accepted 2026-04-21 00:45:24,914 - lava_callback - INFO - Starting async job submit for node 69e425a03551964617338b7f 2026-04-21 00:45:24,915 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:45:24,929 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e425a03551964617338b7f HTTP/1.1" 200 None 2026-04-21 00:45:24,930 - lava_callback - DEBUG - Processing node 69e425a03551964617338b7f: baseline-arm-baylibre 2026-04-21 00:45:24,930 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e425a03551964617338b7f 2026-04-21 00:45:24,931 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:45:24,943 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e425a03551964617338b7f HTTP/1.1" 200 None 2026-04-21 00:45:24,945 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:45:24,958 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 00:45:24,959 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:45:24,976 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e425a03551964617338b7f HTTP/1.1" 200 None 2026-04-21 00:45:24,977 - lava_callback - ERROR - Error processing callback for node 69e425a03551964617338b7f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:37524 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48756 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34378 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50412 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:46:05,033 - lava_callback - INFO - Received callback for node 69e425c43551964617338b8c INFO: 172.20.0.1:39292 - "POST /node/69e425c43551964617338b8c HTTP/1.0" 202 Accepted 2026-04-21 00:46:05,039 - lava_callback - INFO - Starting async job submit for node 69e425c43551964617338b8c 2026-04-21 00:46:05,040 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:46:05,060 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e425c43551964617338b8c HTTP/1.1" 200 None 2026-04-21 00:46:05,061 - lava_callback - DEBUG - Processing node 69e425c43551964617338b8c: baseline-arm-baylibre 2026-04-21 00:46:05,061 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e425c43551964617338b8c 2026-04-21 00:46:05,062 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:46:05,077 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e425c43551964617338b8c HTTP/1.1" 200 None 2026-04-21 00:46:05,079 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:46:05,094 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 00:46:05,096 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:46:05,113 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e425c43551964617338b8c HTTP/1.1" 200 None 2026-04-21 00:46:05,115 - lava_callback - ERROR - Error processing callback for node 69e425c43551964617338b8c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:39304 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60288 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60302 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34226 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34232 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55180 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:46:45,012 - lava_callback - INFO - Received callback for node 69e425e83551964617338b97 INFO: 172.20.0.1:58446 - "POST /node/69e425e83551964617338b97 HTTP/1.0" 202 Accepted 2026-04-21 00:46:45,019 - lava_callback - INFO - Starting async job submit for node 69e425e83551964617338b97 2026-04-21 00:46:45,020 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:46:45,034 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e425e83551964617338b97 HTTP/1.1" 200 None 2026-04-21 00:46:45,035 - lava_callback - DEBUG - Processing node 69e425e83551964617338b97: baseline-arm-baylibre 2026-04-21 00:46:45,036 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e425e83551964617338b97 2026-04-21 00:46:45,036 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:46:45,061 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e425e83551964617338b97 HTTP/1.1" 200 None 2026-04-21 00:46:45,062 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:46:45,076 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 00:46:45,078 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:46:45,096 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e425e83551964617338b97 HTTP/1.1" 200 None 2026-04-21 00:46:45,097 - lava_callback - ERROR - Error processing callback for node 69e425e83551964617338b97: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:58456 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42230 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51152 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58520 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58534 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54030 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:47:25,157 - lava_callback - INFO - Received callback for node 69e4260d3551964617338baa INFO: 172.20.0.1:54038 - "POST /node/69e4260d3551964617338baa HTTP/1.0" 202 Accepted 2026-04-21 00:47:25,163 - lava_callback - INFO - Starting async job submit for node 69e4260d3551964617338baa 2026-04-21 00:47:25,164 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:47:25,178 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4260d3551964617338baa HTTP/1.1" 200 None 2026-04-21 00:47:25,179 - lava_callback - DEBUG - Processing node 69e4260d3551964617338baa: baseline-arm-baylibre 2026-04-21 00:47:25,179 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4260d3551964617338baa 2026-04-21 00:47:25,180 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:47:25,193 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4260d3551964617338baa HTTP/1.1" 200 None 2026-04-21 00:47:25,194 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:47:25,207 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 00:47:25,208 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:47:25,225 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4260d3551964617338baa HTTP/1.1" 200 None 2026-04-21 00:47:25,226 - lava_callback - ERROR - Error processing callback for node 69e4260d3551964617338baa: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54050 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46330 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60874 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:47:52,753 - lava_callback - INFO - Received callback for node 69e426323551964617338bb7 INFO: 172.20.0.1:58566 - "POST /node/69e426323551964617338bb7 HTTP/1.0" 202 Accepted 2026-04-21 00:47:52,758 - lava_callback - INFO - Starting async job submit for node 69e426323551964617338bb7 2026-04-21 00:47:52,759 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:47:52,778 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e426323551964617338bb7 HTTP/1.1" 200 None 2026-04-21 00:47:52,779 - lava_callback - DEBUG - Processing node 69e426323551964617338bb7: baseline-arm-clabbe 2026-04-21 00:47:52,779 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e426323551964617338bb7 2026-04-21 00:47:52,780 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:47:52,793 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e426323551964617338bb7 HTTP/1.1" 200 None 2026-04-21 00:47:52,794 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:47:52,807 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 00:47:52,808 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:47:52,826 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e426323551964617338bb7 HTTP/1.1" 200 None 2026-04-21 00:47:52,827 - lava_callback - ERROR - Error processing callback for node 69e426323551964617338bb7: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:58574 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45232 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38730 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:38738 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:48:19,473 - lava_callback - INFO - Received callback for node 69e6c5c360f5826562c15883 INFO: 172.20.0.1:38752 - "POST /node/69e6c5c360f5826562c15883 HTTP/1.0" 202 Accepted 2026-04-21 00:48:19,480 - lava_callback - INFO - Starting async job submit for node 69e6c5c360f5826562c15883 2026-04-21 00:48:19,505 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:48:19,519 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5c360f5826562c15883 HTTP/1.1" 200 None 2026-04-21 00:48:19,520 - lava_callback - DEBUG - Processing node 69e6c5c360f5826562c15883: ltp-cap_bounds 2026-04-21 00:48:19,854 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:48:19,967 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:48:19,968 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-cap_bounds-69e6c5c360f5826562c15883/log.txt.gz 2026-04-21 00:48:20,186 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.20.0.1:41720 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:48:20,390 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:48:20,392 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-cap_bounds-69e6c5c360f5826562c15883/lava-logs.yaml 2026-04-21 00:48:20,400 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:48:20,561 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:48:20,562 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-cap_bounds-69e6c5c360f5826562c15883/lava_callback.json.gz 2026-04-21 00:48:20,588 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:48:20,602 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5c360f5826562c15883 HTTP/1.1" 200 None 2026-04-21 00:48:20,603 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:48:20,616 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:48:20,617 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:48:20,638 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c5c360f5826562c15883 HTTP/1.1" 200 None 2026-04-21 00:48:20,639 - lava_callback - INFO - Completed processing callback for node 69e6c5c360f5826562c15883 INFO: 172.20.0.1:41724 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:48:32,700 - lava_callback - INFO - Received callback for node 69e426583551964617338bc4 INFO: 172.20.0.1:57500 - "POST /node/69e426583551964617338bc4 HTTP/1.0" 202 Accepted 2026-04-21 00:48:32,706 - lava_callback - INFO - Starting async job submit for node 69e426583551964617338bc4 2026-04-21 00:48:32,707 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:48:32,721 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e426583551964617338bc4 HTTP/1.1" 200 None 2026-04-21 00:48:32,722 - lava_callback - DEBUG - Processing node 69e426583551964617338bc4: baseline-arm-clabbe 2026-04-21 00:48:32,722 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e426583551964617338bc4 2026-04-21 00:48:32,723 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:48:32,736 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e426583551964617338bc4 HTTP/1.1" 200 None 2026-04-21 00:48:32,738 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:48:32,750 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 00:48:32,751 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:48:32,777 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e426583551964617338bc4 HTTP/1.1" 200 None 2026-04-21 00:48:32,778 - lava_callback - ERROR - Error processing callback for node 69e426583551964617338bc4: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-04-21 00:48:35,373 - lava_callback - INFO - Received callback for node 69e6c59060f5826562c157d6 INFO: 172.20.0.1:57506 - "POST /node/69e6c59060f5826562c157d6 HTTP/1.0" 202 Accepted 2026-04-21 00:48:35,380 - lava_callback - INFO - Starting async job submit for node 69e6c59060f5826562c157d6 2026-04-21 00:48:35,442 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:48:35,457 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c59060f5826562c157d6 HTTP/1.1" 200 None 2026-04-21 00:48:35,458 - lava_callback - DEBUG - Processing node 69e6c59060f5826562c157d6: baseline-arm 2026-04-21 00:48:36,090 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:48:36,176 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:48:36,178 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69e6c59060f5826562c157d6/log.txt.gz INFO: 172.20.0.1:57516 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:48:36,541 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:48:36,749 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:48:36,751 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69e6c59060f5826562c157d6/lava-logs.yaml 2026-04-21 00:48:36,766 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:48:36,884 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:48:36,886 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69e6c59060f5826562c157d6/lava_callback.json.gz DEBUG: Setup failed for 69e6c59060f5826562c157d6. Transitting job node result: incomplete -> fail 2026-04-21 00:48:36,994 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:48:37,008 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c59060f5826562c157d6 HTTP/1.1" 200 None 2026-04-21 00:48:37,009 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:48:37,022 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:48:37,023 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:48:37,045 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c59060f5826562c157d6 HTTP/1.1" 200 None 2026-04-21 00:48:37,158 - lava_callback - INFO - Completed processing callback for node 69e6c59060f5826562c157d6 2026-04-21 00:48:43,151 - lava_callback - INFO - Received callback for node 69e6c45b60f5826562c15188 INFO: 172.20.0.1:57040 - "POST /node/69e6c45b60f5826562c15188 HTTP/1.0" 202 Accepted 2026-04-21 00:48:43,158 - lava_callback - INFO - Starting async job submit for node 69e6c45b60f5826562c15188 2026-04-21 00:48:43,222 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:48:43,236 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c45b60f5826562c15188 HTTP/1.1" 200 None 2026-04-21 00:48:43,237 - lava_callback - DEBUG - Processing node 69e6c45b60f5826562c15188: ltp-smoke 2026-04-21 00:48:43,710 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:48:43,790 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:48:43,792 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-smoke-69e6c45b60f5826562c15188/log.txt.gz 2026-04-21 00:48:44,077 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:48:44,399 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:48:44,417 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-04-21 00:48:44,422 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:48:44,424 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-smoke-69e6c45b60f5826562c15188/lava-logs.yaml 2026-04-21 00:48:44,437 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:48:44,613 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:48:44,615 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-smoke-69e6c45b60f5826562c15188/lava_callback.json.gz 2026-04-21 00:48:44,730 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:48:44,744 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c45b60f5826562c15188 HTTP/1.1" 200 None 2026-04-21 00:48:44,746 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:48:44,764 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c21560f5826562c15091 HTTP/1.1" 200 None 2026-04-21 00:48:44,765 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:48:44,790 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c45b60f5826562c15188 HTTP/1.1" 200 None 2026-04-21 00:48:44,890 - lava_callback - INFO - Completed processing callback for node 69e6c45b60f5826562c15188 INFO: 172.20.0.1:57042 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54382 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54674 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:49:12,738 - lava_callback - INFO - Received callback for node 69e4267c3551964617338bd1 INFO: 172.20.0.1:38810 - "POST /node/69e4267c3551964617338bd1 HTTP/1.0" 202 Accepted 2026-04-21 00:49:12,744 - lava_callback - INFO - Starting async job submit for node 69e4267c3551964617338bd1 2026-04-21 00:49:12,745 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:49:12,759 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4267c3551964617338bd1 HTTP/1.1" 200 None 2026-04-21 00:49:12,760 - lava_callback - DEBUG - Processing node 69e4267c3551964617338bd1: baseline-arm-clabbe 2026-04-21 00:49:12,760 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4267c3551964617338bd1 2026-04-21 00:49:12,760 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:49:12,774 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4267c3551964617338bd1 HTTP/1.1" 200 None 2026-04-21 00:49:12,775 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:49:12,787 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 00:49:12,789 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:49:12,810 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4267c3551964617338bd1 HTTP/1.1" 200 None 2026-04-21 00:49:12,811 - lava_callback - ERROR - Error processing callback for node 69e4267c3551964617338bd1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-04-21 00:49:14,419 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:49:14,436 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:38824 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:38840 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59726 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59742 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55020 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57498 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:49:52,729 - lava_callback - INFO - Received callback for node 69e426a23551964617338be0 INFO: 172.20.0.1:51318 - "POST /node/69e426a23551964617338be0 HTTP/1.0" 202 Accepted 2026-04-21 00:49:52,734 - lava_callback - INFO - Starting async job submit for node 69e426a23551964617338be0 2026-04-21 00:49:52,736 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:49:52,750 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e426a23551964617338be0 HTTP/1.1" 200 None 2026-04-21 00:49:52,751 - lava_callback - DEBUG - Processing node 69e426a23551964617338be0: baseline-arm-clabbe 2026-04-21 00:49:52,751 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e426a23551964617338be0 2026-04-21 00:49:52,752 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:49:52,779 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e426a23551964617338be0 HTTP/1.1" 200 None 2026-04-21 00:49:52,781 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:49:52,795 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 00:49:52,796 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:49:52,814 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e426a23551964617338be0 HTTP/1.1" 200 None 2026-04-21 00:49:52,815 - lava_callback - ERROR - Error processing callback for node 69e426a23551964617338be0: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:51320 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33306 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60718 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60722 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34350 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34358 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:50:32,801 - lava_callback - INFO - Received callback for node 69e426c73551964617338beb INFO: 172.20.0.1:41032 - "POST /node/69e426c73551964617338beb HTTP/1.0" 202 Accepted 2026-04-21 00:50:32,811 - lava_callback - INFO - Starting async job submit for node 69e426c73551964617338beb 2026-04-21 00:50:32,812 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:50:32,834 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e426c73551964617338beb HTTP/1.1" 200 None 2026-04-21 00:50:32,835 - lava_callback - DEBUG - Processing node 69e426c73551964617338beb: baseline-arm-clabbe 2026-04-21 00:50:32,835 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e426c73551964617338beb 2026-04-21 00:50:32,836 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:50:32,856 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e426c73551964617338beb HTTP/1.1" 200 None 2026-04-21 00:50:32,858 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:50:32,877 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 00:50:32,878 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:50:32,897 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e426c73551964617338beb HTTP/1.1" 200 None 2026-04-21 00:50:32,898 - lava_callback - ERROR - Error processing callback for node 69e426c73551964617338beb: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:41046 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33118 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56304 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37254 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:51:12,820 - lava_callback - INFO - Received callback for node 69e426eb3551964617338bf8 INFO: 172.20.0.1:59688 - "POST /node/69e426eb3551964617338bf8 HTTP/1.0" 202 Accepted 2026-04-21 00:51:12,826 - lava_callback - INFO - Starting async job submit for node 69e426eb3551964617338bf8 2026-04-21 00:51:12,827 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:51:12,841 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e426eb3551964617338bf8 HTTP/1.1" 200 None 2026-04-21 00:51:12,842 - lava_callback - DEBUG - Processing node 69e426eb3551964617338bf8: baseline-arm-clabbe 2026-04-21 00:51:12,842 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e426eb3551964617338bf8 2026-04-21 00:51:12,843 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:51:12,858 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e426eb3551964617338bf8 HTTP/1.1" 200 None 2026-04-21 00:51:12,860 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:51:12,894 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 00:51:12,896 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:51:12,919 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e426eb3551964617338bf8 HTTP/1.1" 200 None 2026-04-21 00:51:12,921 - lava_callback - ERROR - Error processing callback for node 69e426eb3551964617338bf8: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:59696 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59704 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:51:19,510 - lava_callback - INFO - Received callback for node 69e6c5c560f5826562c15886 INFO: 172.20.0.1:59720 - "POST /node/69e6c5c560f5826562c15886 HTTP/1.0" 202 Accepted 2026-04-21 00:51:19,517 - lava_callback - INFO - Starting async job submit for node 69e6c5c560f5826562c15886 2026-04-21 00:51:19,542 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:51:19,555 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5c560f5826562c15886 HTTP/1.1" 200 None 2026-04-21 00:51:19,556 - lava_callback - DEBUG - Processing node 69e6c5c560f5826562c15886: ltp-containers 2026-04-21 00:51:19,955 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:51:20,057 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:51:20,058 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-containers-69e6c5c560f5826562c15886/log.txt.gz INFO: 172.20.0.1:45464 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:51:20,292 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:51:20,425 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:51:20,427 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-containers-69e6c5c560f5826562c15886/lava-logs.yaml 2026-04-21 00:51:20,436 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:51:20,726 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:51:20,727 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-containers-69e6c5c560f5826562c15886/lava_callback.json.gz 2026-04-21 00:51:20,752 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:51:20,766 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5c560f5826562c15886 HTTP/1.1" 200 None 2026-04-21 00:51:20,767 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:51:20,789 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:51:20,790 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:51:20,812 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c5c560f5826562c15886 HTTP/1.1" 200 None 2026-04-21 00:51:20,813 - lava_callback - INFO - Completed processing callback for node 69e6c5c560f5826562c15886 INFO: 172.20.0.1:45470 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:51:32,828 - lava_callback - INFO - Received callback for node 69e427103551964617338c04 INFO: 172.20.0.1:33506 - "POST /node/69e427103551964617338c04 HTTP/1.0" 202 Accepted 2026-04-21 00:51:32,834 - lava_callback - INFO - Starting async job submit for node 69e427103551964617338c04 2026-04-21 00:51:32,835 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:51:32,850 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e427103551964617338c04 HTTP/1.1" 200 None 2026-04-21 00:51:32,851 - lava_callback - DEBUG - Processing node 69e427103551964617338c04: baseline-arm-clabbe 2026-04-21 00:51:32,851 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e427103551964617338c04 2026-04-21 00:51:32,852 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:51:32,872 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e427103551964617338c04 HTTP/1.1" 200 None 2026-04-21 00:51:32,874 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:51:32,888 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 00:51:32,889 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:51:32,910 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e427103551964617338c04 HTTP/1.1" 200 None 2026-04-21 00:51:32,911 - lava_callback - ERROR - Error processing callback for node 69e427103551964617338c04: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:33508 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:51:44,438 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:51:44,454 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:43680 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54144 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:52:02,033 - lava_callback - INFO - Received callback for node 69e6c5c660f5826562c1588a INFO: 172.20.0.1:59838 - "POST /node/69e6c5c660f5826562c1588a HTTP/1.0" 202 Accepted 2026-04-21 00:52:02,040 - lava_callback - INFO - Starting async job submit for node 69e6c5c660f5826562c1588a 2026-04-21 00:52:02,065 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:52:02,080 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5c660f5826562c1588a HTTP/1.1" 200 None 2026-04-21 00:52:02,081 - lava_callback - DEBUG - Processing node 69e6c5c660f5826562c1588a: ltp-controllers 2026-04-21 00:52:02,501 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:52:02,622 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:52:02,624 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-controllers-69e6c5c660f5826562c1588a/log.txt.gz 2026-04-21 00:52:02,888 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:52:03,041 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:52:03,043 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-controllers-69e6c5c660f5826562c1588a/lava-logs.yaml 2026-04-21 00:52:03,054 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:52:03,326 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:52:03,328 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-controllers-69e6c5c660f5826562c1588a/lava_callback.json.gz 2026-04-21 00:52:03,353 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:52:03,366 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5c660f5826562c1588a HTTP/1.1" 200 None 2026-04-21 00:52:03,368 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:52:03,381 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:52:03,382 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:52:03,408 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c5c660f5826562c1588a HTTP/1.1" 200 None 2026-04-21 00:52:03,410 - lava_callback - INFO - Completed processing callback for node 69e6c5c660f5826562c1588a INFO: 172.20.0.1:59850 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:52:12,844 - lava_callback - INFO - Received callback for node 69e427353551964617338c13 INFO: 172.20.0.1:55686 - "POST /node/69e427353551964617338c13 HTTP/1.0" 202 Accepted 2026-04-21 00:52:12,851 - lava_callback - INFO - Starting async job submit for node 69e427353551964617338c13 2026-04-21 00:52:12,852 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:52:12,870 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e427353551964617338c13 HTTP/1.1" 200 None 2026-04-21 00:52:12,871 - lava_callback - DEBUG - Processing node 69e427353551964617338c13: baseline-arm-clabbe 2026-04-21 00:52:12,872 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e427353551964617338c13 2026-04-21 00:52:12,872 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:52:12,893 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e427353551964617338c13 HTTP/1.1" 200 None 2026-04-21 00:52:12,894 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:52:12,908 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 00:52:12,909 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:52:12,943 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e427353551964617338c13 HTTP/1.1" 200 None 2026-04-21 00:52:12,945 - lava_callback - ERROR - Error processing callback for node 69e427353551964617338c13: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-04-21 00:52:14,456 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:52:14,478 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:55692 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:55706 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:52:19,662 - lava_callback - INFO - Received callback for node 69e6c5c960f5826562c158aa INFO: 172.20.0.1:55716 - "POST /node/69e6c5c960f5826562c158aa HTTP/1.0" 202 Accepted 2026-04-21 00:52:19,668 - lava_callback - INFO - Starting async job submit for node 69e6c5c960f5826562c158aa 2026-04-21 00:52:19,693 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:52:19,729 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5c960f5826562c158aa HTTP/1.1" 200 None 2026-04-21 00:52:19,730 - lava_callback - DEBUG - Processing node 69e6c5c960f5826562c158aa: ltp-cve 2026-04-21 00:52:20,123 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.20.0.1:48366 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:52:20,408 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:52:20,409 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-cve-69e6c5c960f5826562c158aa/log.txt.gz 2026-04-21 00:52:20,614 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:52:20,782 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:52:20,784 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-cve-69e6c5c960f5826562c158aa/lava-logs.yaml 2026-04-21 00:52:20,793 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:52:20,941 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:52:20,942 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-cve-69e6c5c960f5826562c158aa/lava_callback.json.gz 2026-04-21 00:52:20,967 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:52:20,981 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5c960f5826562c158aa HTTP/1.1" 200 None 2026-04-21 00:52:20,982 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:52:20,995 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:52:20,996 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:52:21,024 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c5c960f5826562c158aa HTTP/1.1" 200 None 2026-04-21 00:52:21,025 - lava_callback - INFO - Completed processing callback for node 69e6c5c960f5826562c158aa INFO: 172.20.0.1:48378 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38672 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:52:39,865 - lava_callback - INFO - Received callback for node 69e6c5cd60f5826562c158b6 INFO: 172.20.0.1:55576 - "POST /node/69e6c5cd60f5826562c158b6 HTTP/1.0" 202 Accepted 2026-04-21 00:52:39,871 - lava_callback - INFO - Starting async job submit for node 69e6c5cd60f5826562c158b6 2026-04-21 00:52:39,897 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:52:39,911 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5cd60f5826562c158b6 HTTP/1.1" 200 None 2026-04-21 00:52:39,912 - lava_callback - DEBUG - Processing node 69e6c5cd60f5826562c158b6: ltp-fs_bind 2026-04-21 00:52:40,290 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:52:40,373 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:52:40,375 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-fs_bind-69e6c5cd60f5826562c158b6/log.txt.gz 2026-04-21 00:52:40,614 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:52:40,770 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:52:40,772 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-fs_bind-69e6c5cd60f5826562c158b6/lava-logs.yaml 2026-04-21 00:52:40,780 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:52:40,904 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:52:40,906 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-fs_bind-69e6c5cd60f5826562c158b6/lava_callback.json.gz 2026-04-21 00:52:40,931 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:52:40,945 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5cd60f5826562c158b6 HTTP/1.1" 200 None 2026-04-21 00:52:40,946 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:52:40,959 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:52:40,960 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:52:40,980 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c5cd60f5826562c158b6 HTTP/1.1" 200 None 2026-04-21 00:52:40,982 - lava_callback - INFO - Completed processing callback for node 69e6c5cd60f5826562c158b6 2026-04-21 00:52:44,481 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:52:44,496 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:55586 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:52:52,943 - lava_callback - INFO - Received callback for node 69e427593551964617338c1d INFO: 172.20.0.1:36838 - "POST /node/69e427593551964617338c1d HTTP/1.0" 202 Accepted 2026-04-21 00:52:52,953 - lava_callback - INFO - Starting async job submit for node 69e427593551964617338c1d 2026-04-21 00:52:52,954 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:52:52,968 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e427593551964617338c1d HTTP/1.1" 200 None 2026-04-21 00:52:52,969 - lava_callback - DEBUG - Processing node 69e427593551964617338c1d: baseline-arm-clabbe 2026-04-21 00:52:52,969 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e427593551964617338c1d 2026-04-21 00:52:52,970 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:52:52,983 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e427593551964617338c1d HTTP/1.1" 200 None 2026-04-21 00:52:52,984 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:52:53,005 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 00:52:53,007 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:52:53,034 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e427593551964617338c1d HTTP/1.1" 200 None 2026-04-21 00:52:53,035 - lava_callback - ERROR - Error processing callback for node 69e427593551964617338c1d: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:36844 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34038 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34002 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34010 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34440 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34452 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45608 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44446 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55380 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43310 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34552 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34554 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49008 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49012 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35446 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59678 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60838 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58370 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57838 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57850 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59482 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59486 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58470 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:55:39,736 - lava_callback - INFO - Received callback for node 69e6c5ce60f5826562c158b9 INFO: 172.20.0.1:57042 - "POST /node/69e6c5ce60f5826562c158b9 HTTP/1.0" 202 Accepted 2026-04-21 00:55:39,743 - lava_callback - INFO - Starting async job submit for node 69e6c5ce60f5826562c158b9 2026-04-21 00:55:39,768 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:55:39,781 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5ce60f5826562c158b9 HTTP/1.1" 200 None 2026-04-21 00:55:39,782 - lava_callback - DEBUG - Processing node 69e6c5ce60f5826562c158b9: ltp-fsx 2026-04-21 00:55:40,136 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:55:40,249 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:55:40,250 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-fsx-69e6c5ce60f5826562c158b9/log.txt.gz 2026-04-21 00:55:40,452 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:55:40,622 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:55:40,624 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-fsx-69e6c5ce60f5826562c158b9/lava-logs.yaml 2026-04-21 00:55:40,633 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:55:40,770 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:55:40,772 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-fsx-69e6c5ce60f5826562c158b9/lava_callback.json.gz 2026-04-21 00:55:40,797 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:55:40,811 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5ce60f5826562c158b9 HTTP/1.1" 200 None 2026-04-21 00:55:40,812 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:55:40,825 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:55:40,827 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:55:40,852 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c5ce60f5826562c158b9 HTTP/1.1" 200 None 2026-04-21 00:55:40,853 - lava_callback - INFO - Completed processing callback for node 69e6c5ce60f5826562c158b9 2026-04-21 00:55:44,499 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:55:44,528 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:57054 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60380 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:56:02,100 - lava_callback - INFO - Received callback for node 69e6c5cf60f5826562c158bc INFO: 172.20.0.1:53100 - "POST /node/69e6c5cf60f5826562c158bc HTTP/1.0" 202 Accepted 2026-04-21 00:56:02,106 - lava_callback - INFO - Starting async job submit for node 69e6c5cf60f5826562c158bc 2026-04-21 00:56:02,132 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:56:02,146 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5cf60f5826562c158bc HTTP/1.1" 200 None 2026-04-21 00:56:02,147 - lava_callback - DEBUG - Processing node 69e6c5cf60f5826562c158bc: ltp-input 2026-04-21 00:56:02,508 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:56:02,623 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:56:02,624 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-input-69e6c5cf60f5826562c158bc/log.txt.gz 2026-04-21 00:56:02,826 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:56:02,997 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:56:03,000 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-input-69e6c5cf60f5826562c158bc/lava-logs.yaml 2026-04-21 00:56:03,008 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:56:03,113 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:56:03,114 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-input-69e6c5cf60f5826562c158bc/lava_callback.json.gz 2026-04-21 00:56:03,139 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:56:03,154 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5cf60f5826562c158bc HTTP/1.1" 200 None 2026-04-21 00:56:03,156 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:56:03,175 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:56:03,177 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:56:03,208 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c5cf60f5826562c158bc HTTP/1.1" 200 None 2026-04-21 00:56:03,209 - lava_callback - INFO - Completed processing callback for node 69e6c5cf60f5826562c158bc 2026-04-21 00:56:05,714 - lava_callback - INFO - Received callback for node 69e428183551964617338c32 INFO: 172.20.0.1:53102 - "POST /node/69e428183551964617338c32 HTTP/1.0" 202 Accepted 2026-04-21 00:56:05,719 - lava_callback - INFO - Starting async job submit for node 69e428183551964617338c32 2026-04-21 00:56:05,720 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:56:05,734 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e428183551964617338c32 HTTP/1.1" 200 None 2026-04-21 00:56:05,735 - lava_callback - DEBUG - Processing node 69e428183551964617338c32: baseline-arm-baylibre 2026-04-21 00:56:05,735 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e428183551964617338c32 2026-04-21 00:56:05,736 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:56:05,749 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e428183551964617338c32 HTTP/1.1" 200 None 2026-04-21 00:56:05,750 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:56:05,769 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc36005dbfa74e32f9687c HTTP/1.1" 200 None 2026-04-21 00:56:05,771 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:56:05,790 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e428183551964617338c32 HTTP/1.1" 200 None 2026-04-21 00:56:05,791 - lava_callback - ERROR - Error processing callback for node 69e428183551964617338c32: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:53118 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:56:14,530 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:56:14,546 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:42852 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:42864 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34428 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34436 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35890 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:56:39,755 - lava_callback - INFO - Received callback for node 69e6c5d260f5826562c158c2 INFO: 172.20.0.1:47026 - "POST /node/69e6c5d260f5826562c158c2 HTTP/1.0" 202 Accepted 2026-04-21 00:56:39,761 - lava_callback - INFO - Starting async job submit for node 69e6c5d260f5826562c158c2 2026-04-21 00:56:39,786 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:56:39,800 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5d260f5826562c158c2 HTTP/1.1" 200 None 2026-04-21 00:56:39,801 - lava_callback - DEBUG - Processing node 69e6c5d260f5826562c158c2: ltp-pty 2026-04-21 00:56:40,139 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:56:40,238 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:56:40,240 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-pty-69e6c5d260f5826562c158c2/log.txt.gz 2026-04-21 00:56:40,473 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:56:40,630 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:56:40,633 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-pty-69e6c5d260f5826562c158c2/lava-logs.yaml 2026-04-21 00:56:40,642 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:56:40,775 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:56:40,777 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-pty-69e6c5d260f5826562c158c2/lava_callback.json.gz 2026-04-21 00:56:40,803 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:56:40,818 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5d260f5826562c158c2 HTTP/1.1" 200 None 2026-04-21 00:56:40,819 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:56:40,832 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:56:40,834 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:56:40,857 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c5d260f5826562c158c2 HTTP/1.1" 200 None 2026-04-21 00:56:40,858 - lava_callback - INFO - Completed processing callback for node 69e6c5d260f5826562c158c2 2026-04-21 00:56:44,548 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:56:44,564 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-04-21 00:56:45,549 - lava_callback - INFO - Received callback for node 69e428413551964617338c42 INFO: 172.20.0.1:47042 - "POST /node/69e428413551964617338c42 HTTP/1.0" 202 Accepted 2026-04-21 00:56:45,554 - lava_callback - INFO - Starting async job submit for node 69e428413551964617338c42 2026-04-21 00:56:45,555 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:56:45,569 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e428413551964617338c42 HTTP/1.1" 200 None 2026-04-21 00:56:45,570 - lava_callback - DEBUG - Processing node 69e428413551964617338c42: baseline-arm-baylibre 2026-04-21 00:56:45,570 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e428413551964617338c42 2026-04-21 00:56:45,571 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:56:45,584 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e428413551964617338c42 HTTP/1.1" 200 None 2026-04-21 00:56:45,585 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:56:45,599 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc36005dbfa74e32f9687c HTTP/1.1" 200 None 2026-04-21 00:56:45,600 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:56:45,618 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e428413551964617338c42 HTTP/1.1" 200 None 2026-04-21 00:56:45,619 - lava_callback - ERROR - Error processing callback for node 69e428413551964617338c42: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47046 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41896 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34582 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51666 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51678 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42968 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:57:25,681 - lava_callback - INFO - Received callback for node 69e428693551964617338c4e INFO: 172.20.0.1:42980 - "POST /node/69e428693551964617338c4e HTTP/1.0" 202 Accepted 2026-04-21 00:57:25,689 - lava_callback - INFO - Starting async job submit for node 69e428693551964617338c4e 2026-04-21 00:57:25,691 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:57:25,706 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e428693551964617338c4e HTTP/1.1" 200 None 2026-04-21 00:57:25,707 - lava_callback - DEBUG - Processing node 69e428693551964617338c4e: baseline-arm-baylibre 2026-04-21 00:57:25,707 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e428693551964617338c4e 2026-04-21 00:57:25,708 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:57:25,721 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e428693551964617338c4e HTTP/1.1" 200 None 2026-04-21 00:57:25,722 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:57:25,735 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc36005dbfa74e32f9687c HTTP/1.1" 200 None 2026-04-21 00:57:25,736 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:57:25,753 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e428693551964617338c4e HTTP/1.1" 200 None 2026-04-21 00:57:25,754 - lava_callback - ERROR - Error processing callback for node 69e428693551964617338c4e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:42990 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47314 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52054 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51888 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:57:57,372 - lava_callback - INFO - Received callback for node 69e6cb8560f5826562c15c10 INFO: 172.20.0.1:51902 - "POST /node/69e6cb8560f5826562c15c10 HTTP/1.0" 202 Accepted 2026-04-21 00:57:57,378 - lava_callback - INFO - Starting async job submit for node 69e6cb8560f5826562c15c10 2026-04-21 00:57:57,381 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:57:57,394 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb8560f5826562c15c10 HTTP/1.1" 200 None 2026-04-21 00:57:57,395 - lava_callback - DEBUG - Processing node 69e6cb8560f5826562c15c10: baseline-nfs-arm64 2026-04-21 00:57:57,397 - lava_callback - WARNING - Failed to upload log 2026-04-21 00:57:57,398 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:57:57,482 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:57:57,484 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb8560f5826562c15c10/lava-logs.yaml 2026-04-21 00:57:57,486 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:57:57,612 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:57:57,613 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb8560f5826562c15c10/lava_callback.json.gz 2026-04-21 00:57:57,615 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:57:57,630 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb8560f5826562c15c10 HTTP/1.1" 200 None 2026-04-21 00:57:57,632 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:57:57,645 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 00:57:57,647 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:57:57,664 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb8560f5826562c15c10 HTTP/1.1" 200 None 2026-04-21 00:57:57,667 - lava_callback - INFO - Completed processing callback for node 69e6cb8560f5826562c15c10 2026-04-21 00:58:05,669 - lava_callback - INFO - Received callback for node 69e428933551964617338c5b INFO: 172.20.0.1:47072 - "POST /node/69e428933551964617338c5b HTTP/1.0" 202 Accepted 2026-04-21 00:58:05,678 - lava_callback - INFO - Starting async job submit for node 69e428933551964617338c5b 2026-04-21 00:58:05,680 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:05,701 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e428933551964617338c5b HTTP/1.1" 200 None 2026-04-21 00:58:05,702 - lava_callback - DEBUG - Processing node 69e428933551964617338c5b: baseline-arm-baylibre 2026-04-21 00:58:05,702 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e428933551964617338c5b 2026-04-21 00:58:05,703 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:05,724 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e428933551964617338c5b HTTP/1.1" 200 None 2026-04-21 00:58:05,726 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:05,747 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc36005dbfa74e32f9687c HTTP/1.1" 200 None 2026-04-21 00:58:05,750 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:05,796 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e428933551964617338c5b HTTP/1.1" 200 None 2026-04-21 00:58:05,797 - lava_callback - ERROR - Error processing callback for node 69e428933551964617338c5b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47078 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:58:09,726 - lava_callback - INFO - Received callback for node 69e6cb9860f5826562c15c66 INFO: 172.20.0.1:37416 - "POST /node/69e6cb9860f5826562c15c66 HTTP/1.0" 202 Accepted 2026-04-21 00:58:09,734 - lava_callback - INFO - Starting async job submit for node 69e6cb9860f5826562c15c66 2026-04-21 00:58:09,737 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:09,751 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb9860f5826562c15c66 HTTP/1.1" 200 None 2026-04-21 00:58:09,752 - lava_callback - DEBUG - Processing node 69e6cb9860f5826562c15c66: baseline-arm 2026-04-21 00:58:09,757 - lava_callback - WARNING - Failed to upload log 2026-04-21 00:58:09,760 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:09,853 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:09,855 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69e6cb9860f5826562c15c66/lava-logs.yaml 2026-04-21 00:58:09,856 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:09,926 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:09,928 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69e6cb9860f5826562c15c66/lava_callback.json.gz 2026-04-21 00:58:09,931 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:09,944 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb9860f5826562c15c66 HTTP/1.1" 200 None 2026-04-21 00:58:09,946 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:09,962 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:58:09,964 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:09,981 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb9860f5826562c15c66 HTTP/1.1" 200 None 2026-04-21 00:58:09,990 - lava_callback - INFO - Completed processing callback for node 69e6cb9860f5826562c15c66 2026-04-21 00:58:14,566 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:14,582 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:37430 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37432 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60850 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60866 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:58:28,129 - lava_callback - INFO - Received callback for node 69e6cb7460f5826562c15bdb INFO: 172.20.0.1:60868 - "POST /node/69e6cb7460f5826562c15bdb HTTP/1.0" 202 Accepted 2026-04-21 00:58:28,136 - lava_callback - INFO - Starting async job submit for node 69e6cb7460f5826562c15bdb 2026-04-21 00:58:28,158 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:28,173 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7460f5826562c15bdb HTTP/1.1" 200 None 2026-04-21 00:58:28,173 - lava_callback - DEBUG - Processing node 69e6cb7460f5826562c15bdb: baseline-arm64 2026-04-21 00:58:28,539 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:28,685 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:28,687 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7460f5826562c15bdb/log.txt.gz 2026-04-21 00:58:28,812 - lava_callback - INFO - Received callback for node 69e6cb6b60f5826562c15bc3 2026-04-21 00:58:28,827 - lava_callback - INFO - Starting async job submit for node 69e6cb6b60f5826562c15bc3 INFO: 172.20.0.1:60882 - "POST /node/69e6cb6b60f5826562c15bc3 HTTP/1.0" 202 Accepted 2026-04-21 00:58:28,868 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:28,928 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:28,930 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb6b60f5826562c15bc3 HTTP/1.1" 200 None 2026-04-21 00:58:28,931 - lava_callback - DEBUG - Processing node 69e6cb6b60f5826562c15bc3: baseline-arm64 2026-04-21 00:58:29,183 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:29,185 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7460f5826562c15bdb/lava-logs.yaml 2026-04-21 00:58:29,187 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:29,193 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:29,299 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:29,300 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb6b60f5826562c15bc3/log.txt.gz 2026-04-21 00:58:29,317 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:29,350 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7460f5826562c15bdb/lava_callback.json.gz 2026-04-21 00:58:29,413 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:29,443 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:29,449 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7460f5826562c15bdb HTTP/1.1" 200 None 2026-04-21 00:58:29,452 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:29,479 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 00:58:29,481 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:29,516 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb7460f5826562c15bdb HTTP/1.1" 200 None 2026-04-21 00:58:29,518 - lava_callback - INFO - Completed processing callback for node 69e6cb7460f5826562c15bdb 2026-04-21 00:58:29,634 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:29,636 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb6b60f5826562c15bc3/lava-logs.yaml 2026-04-21 00:58:29,642 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:29,777 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:29,778 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb6b60f5826562c15bc3/lava_callback.json.gz 2026-04-21 00:58:29,798 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:29,815 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb6b60f5826562c15bc3 HTTP/1.1" 200 None 2026-04-21 00:58:29,817 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:29,830 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 00:58:29,832 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:29,860 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb6b60f5826562c15bc3 HTTP/1.1" 200 None 2026-04-21 00:58:29,861 - lava_callback - INFO - Completed processing callback for node 69e6cb6b60f5826562c15bc3 2026-04-21 00:58:31,626 - lava_callback - INFO - Received callback for node 69e6cb7560f5826562c15bde INFO: 172.20.0.1:60812 - "POST /node/69e6cb7560f5826562c15bde HTTP/1.0" 202 Accepted 2026-04-21 00:58:31,633 - lava_callback - INFO - Starting async job submit for node 69e6cb7560f5826562c15bde 2026-04-21 00:58:31,654 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:31,668 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7560f5826562c15bde HTTP/1.1" 200 None 2026-04-21 00:58:31,668 - lava_callback - DEBUG - Processing node 69e6cb7560f5826562c15bde: baseline-arm64 2026-04-21 00:58:32,053 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:32,196 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:32,198 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7560f5826562c15bde/log.txt.gz 2026-04-21 00:58:32,298 - lava_callback - INFO - Received callback for node 69e6cb6d60f5826562c15bc9 INFO: 172.20.0.1:60826 - "POST /node/69e6cb6d60f5826562c15bc9 HTTP/1.0" 202 Accepted 2026-04-21 00:58:32,317 - lava_callback - INFO - Starting async job submit for node 69e6cb6d60f5826562c15bc9 2026-04-21 00:58:32,376 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:32,441 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb6d60f5826562c15bc9 HTTP/1.1" 200 None 2026-04-21 00:58:32,443 - lava_callback - DEBUG - Processing node 69e6cb6d60f5826562c15bc9: baseline-arm64 2026-04-21 00:58:32,464 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:32,722 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:32,749 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7560f5826562c15bde/lava-logs.yaml 2026-04-21 00:58:32,751 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:32,757 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:32,823 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:32,825 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb6d60f5826562c15bc9/log.txt.gz 2026-04-21 00:58:32,879 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:32,914 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7560f5826562c15bde/lava_callback.json.gz 2026-04-21 00:58:32,962 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:33,037 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:33,038 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7560f5826562c15bde HTTP/1.1" 200 None 2026-04-21 00:58:33,041 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:33,060 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 00:58:33,062 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:33,100 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb7560f5826562c15bde HTTP/1.1" 200 None 2026-04-21 00:58:33,107 - lava_callback - INFO - Completed processing callback for node 69e6cb7560f5826562c15bde 2026-04-21 00:58:33,219 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:33,220 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb6d60f5826562c15bc9/lava-logs.yaml 2026-04-21 00:58:33,228 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:33,353 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:33,355 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb6d60f5826562c15bc9/lava_callback.json.gz 2026-04-21 00:58:33,377 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:33,390 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb6d60f5826562c15bc9 HTTP/1.1" 200 None 2026-04-21 00:58:33,392 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:33,405 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 00:58:33,407 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:33,442 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb6d60f5826562c15bc9 HTTP/1.1" 200 None 2026-04-21 00:58:33,444 - lava_callback - INFO - Completed processing callback for node 69e6cb6d60f5826562c15bc9 2026-04-21 00:58:34,534 - lava_callback - INFO - Received callback for node 69e6cb7760f5826562c15be8 INFO: 172.20.0.1:60834 - "POST /node/69e6cb7760f5826562c15be8 HTTP/1.0" 202 Accepted 2026-04-21 00:58:34,540 - lava_callback - INFO - Starting async job submit for node 69e6cb7760f5826562c15be8 2026-04-21 00:58:34,566 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:34,584 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7760f5826562c15be8 HTTP/1.1" 200 None 2026-04-21 00:58:34,585 - lava_callback - DEBUG - Processing node 69e6cb7760f5826562c15be8: baseline-arm64 2026-04-21 00:58:34,765 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:34,836 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:34,837 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7760f5826562c15be8/log.txt.gz 2026-04-21 00:58:34,931 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:35,066 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:35,068 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7760f5826562c15be8/lava-logs.yaml 2026-04-21 00:58:35,073 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:35,221 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:35,222 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7760f5826562c15be8/lava_callback.json.gz 2026-04-21 00:58:35,240 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:35,261 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7760f5826562c15be8 HTTP/1.1" 200 None 2026-04-21 00:58:35,263 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:35,275 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 00:58:35,277 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:35,298 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb7760f5826562c15be8 HTTP/1.1" 200 None 2026-04-21 00:58:35,299 - lava_callback - INFO - Completed processing callback for node 69e6cb7760f5826562c15be8 INFO: 172.20.0.1:60844 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:58:37,848 - lava_callback - INFO - Received callback for node 69e6cb6f60f5826562c15bcf INFO: 172.20.0.1:60856 - "POST /node/69e6cb6f60f5826562c15bcf HTTP/1.0" 202 Accepted 2026-04-21 00:58:37,859 - lava_callback - INFO - Starting async job submit for node 69e6cb6f60f5826562c15bcf 2026-04-21 00:58:37,899 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:37,917 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb6f60f5826562c15bcf HTTP/1.1" 200 None 2026-04-21 00:58:37,918 - lava_callback - DEBUG - Processing node 69e6cb6f60f5826562c15bcf: baseline-arm64 2026-04-21 00:58:38,058 - lava_callback - WARNING - Failed to upload log 2026-04-21 00:58:38,135 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:38,235 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:38,237 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb6f60f5826562c15bcf/lava-logs.yaml 2026-04-21 00:58:38,241 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:38,317 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:38,319 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb6f60f5826562c15bcf/lava_callback.json.gz 2026-04-21 00:58:38,349 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:38,366 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb6f60f5826562c15bcf HTTP/1.1" 200 None 2026-04-21 00:58:38,368 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:38,380 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 00:58:38,381 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:38,399 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb6f60f5826562c15bcf HTTP/1.1" 200 None 2026-04-21 00:58:38,469 - lava_callback - INFO - Completed processing callback for node 69e6cb6f60f5826562c15bcf 2026-04-21 00:58:38,548 - lava_callback - INFO - Received callback for node 69e6cb7260f5826562c15bd8 INFO: 172.20.0.1:60870 - "POST /node/69e6cb7260f5826562c15bd8 HTTP/1.0" 202 Accepted 2026-04-21 00:58:38,557 - lava_callback - INFO - Starting async job submit for node 69e6cb7260f5826562c15bd8 2026-04-21 00:58:38,583 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:38,596 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7260f5826562c15bd8 HTTP/1.1" 200 None 2026-04-21 00:58:38,597 - lava_callback - DEBUG - Processing node 69e6cb7260f5826562c15bd8: baseline-arm64 2026-04-21 00:58:38,866 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:38,936 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:38,937 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7260f5826562c15bd8/log.txt.gz 2026-04-21 00:58:39,098 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:39,225 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:39,227 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7260f5826562c15bd8/lava-logs.yaml 2026-04-21 00:58:39,234 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:39,323 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:39,324 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7260f5826562c15bd8/lava_callback.json.gz 2026-04-21 00:58:39,348 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:39,362 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7260f5826562c15bd8 HTTP/1.1" 200 None 2026-04-21 00:58:39,364 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:39,378 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 00:58:39,380 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:39,404 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb7260f5826562c15bd8 HTTP/1.1" 200 None 2026-04-21 00:58:39,406 - lava_callback - INFO - Completed processing callback for node 69e6cb7260f5826562c15bd8 2026-04-21 00:58:44,584 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:44,607 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 15 2026-04-21 00:58:45,667 - lava_callback - INFO - Received callback for node 69e428bb3551964617338c6a INFO: 172.20.0.1:56612 - "POST /node/69e428bb3551964617338c6a HTTP/1.0" 202 Accepted 2026-04-21 00:58:45,672 - lava_callback - INFO - Starting async job submit for node 69e428bb3551964617338c6a 2026-04-21 00:58:45,673 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:45,687 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e428bb3551964617338c6a HTTP/1.1" 200 None 2026-04-21 00:58:45,688 - lava_callback - DEBUG - Processing node 69e428bb3551964617338c6a: baseline-arm-baylibre 2026-04-21 00:58:45,688 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e428bb3551964617338c6a 2026-04-21 00:58:45,689 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:45,702 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e428bb3551964617338c6a HTTP/1.1" 200 None 2026-04-21 00:58:45,703 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:45,716 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc36005dbfa74e32f9687c HTTP/1.1" 200 None 2026-04-21 00:58:45,718 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:45,748 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e428bb3551964617338c6a HTTP/1.1" 200 None 2026-04-21 00:58:45,749 - lava_callback - ERROR - Error processing callback for node 69e428bb3551964617338c6a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-04-21 00:58:46,226 - lava_callback - INFO - Received callback for node 69e6cb7060f5826562c15bd2 INFO: 172.20.0.1:56624 - "POST /node/69e6cb7060f5826562c15bd2 HTTP/1.0" 202 Accepted 2026-04-21 00:58:46,233 - lava_callback - INFO - Starting async job submit for node 69e6cb7060f5826562c15bd2 2026-04-21 00:58:46,263 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:46,278 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7060f5826562c15bd2 HTTP/1.1" 200 None 2026-04-21 00:58:46,279 - lava_callback - DEBUG - Processing node 69e6cb7060f5826562c15bd2: baseline-arm64 INFO: 172.20.0.1:56632 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:58:46,530 - lava_callback - WARNING - Failed to upload log 2026-04-21 00:58:46,667 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:46,797 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:46,800 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7060f5826562c15bd2/lava-logs.yaml 2026-04-21 00:58:46,806 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:46,936 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:46,938 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7060f5826562c15bd2/lava_callback.json.gz 2026-04-21 00:58:46,986 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:47,016 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7060f5826562c15bd2 HTTP/1.1" 200 None 2026-04-21 00:58:47,019 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:47,041 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 00:58:47,046 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:47,081 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb7060f5826562c15bd2 HTTP/1.1" 200 None 2026-04-21 00:58:47,165 - lava_callback - INFO - Completed processing callback for node 69e6cb7060f5826562c15bd2 INFO: 172.20.0.1:53098 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:58:57,202 - lava_callback - INFO - Received callback for node 69e6cb6c60f5826562c15bc6 INFO: 172.20.0.1:53106 - "POST /node/69e6cb6c60f5826562c15bc6 HTTP/1.0" 202 Accepted 2026-04-21 00:58:57,209 - lava_callback - INFO - Starting async job submit for node 69e6cb6c60f5826562c15bc6 2026-04-21 00:58:57,230 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:57,245 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb6c60f5826562c15bc6 HTTP/1.1" 200 None 2026-04-21 00:58:57,246 - lava_callback - DEBUG - Processing node 69e6cb6c60f5826562c15bc6: baseline-arm64 2026-04-21 00:58:57,483 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:57,559 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:57,561 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb6c60f5826562c15bc6/log.txt.gz 2026-04-21 00:58:57,698 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:57,822 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:57,824 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb6c60f5826562c15bc6/lava-logs.yaml 2026-04-21 00:58:57,831 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:58:57,917 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:58:57,918 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb6c60f5826562c15bc6/lava_callback.json.gz 2026-04-21 00:58:57,940 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:57,954 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb6c60f5826562c15bc6 HTTP/1.1" 200 None 2026-04-21 00:58:57,956 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:57,969 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 00:58:57,970 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:58:57,990 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb6c60f5826562c15bc6 HTTP/1.1" 200 None 2026-04-21 00:58:57,991 - lava_callback - INFO - Completed processing callback for node 69e6cb6c60f5826562c15bc6 2026-04-21 00:59:00,570 - lava_callback - INFO - Received callback for node 69e6cb8360f5826562c15c0a INFO: 172.20.0.1:49750 - "POST /node/69e6cb8360f5826562c15c0a HTTP/1.0" 202 Accepted 2026-04-21 00:59:00,581 - lava_callback - INFO - Starting async job submit for node 69e6cb8360f5826562c15c0a 2026-04-21 00:59:00,605 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:00,619 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb8360f5826562c15c0a HTTP/1.1" 200 None 2026-04-21 00:59:00,619 - lava_callback - DEBUG - Processing node 69e6cb8360f5826562c15c0a: baseline-nfs-arm64 2026-04-21 00:59:01,065 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:01,161 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:01,163 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb8360f5826562c15c0a/log.txt.gz 2026-04-21 00:59:01,436 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:01,576 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:01,579 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb8360f5826562c15c0a/lava-logs.yaml 2026-04-21 00:59:01,589 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:01,691 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:01,693 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb8360f5826562c15c0a/lava_callback.json.gz 2026-04-21 00:59:01,716 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:01,730 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb8360f5826562c15c0a HTTP/1.1" 200 None 2026-04-21 00:59:01,731 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:01,747 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 00:59:01,749 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:01,770 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb8360f5826562c15c0a HTTP/1.1" 200 None 2026-04-21 00:59:01,771 - lava_callback - INFO - Completed processing callback for node 69e6cb8360f5826562c15c0a INFO: 172.20.0.1:49766 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:59:09,042 - lava_callback - INFO - Received callback for node 69e6cb7860f5826562c15beb INFO: 172.20.0.1:49770 - "POST /node/69e6cb7860f5826562c15beb HTTP/1.0" 202 Accepted 2026-04-21 00:59:09,048 - lava_callback - INFO - Starting async job submit for node 69e6cb7860f5826562c15beb 2026-04-21 00:59:09,077 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:09,095 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7860f5826562c15beb HTTP/1.1" 200 None 2026-04-21 00:59:09,096 - lava_callback - DEBUG - Processing node 69e6cb7860f5826562c15beb: baseline-arm64 2026-04-21 00:59:09,332 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:09,411 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:09,412 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7860f5826562c15beb/log.txt.gz 2026-04-21 00:59:09,559 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:09,688 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:09,690 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7860f5826562c15beb/lava-logs.yaml 2026-04-21 00:59:09,696 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:09,814 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:09,816 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7860f5826562c15beb/lava_callback.json.gz 2026-04-21 00:59:09,841 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:09,854 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7860f5826562c15beb HTTP/1.1" 200 None 2026-04-21 00:59:09,856 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:09,869 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 00:59:09,870 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:09,901 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb7860f5826562c15beb HTTP/1.1" 200 None 2026-04-21 00:59:09,902 - lava_callback - INFO - Completed processing callback for node 69e6cb7860f5826562c15beb 2026-04-21 00:59:12,951 - lava_callback - INFO - Received callback for node 69e6cb6e60f5826562c15bcc INFO: 172.20.0.1:50394 - "POST /node/69e6cb6e60f5826562c15bcc HTTP/1.0" 202 Accepted 2026-04-21 00:59:12,957 - lava_callback - INFO - Starting async job submit for node 69e6cb6e60f5826562c15bcc 2026-04-21 00:59:12,979 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:12,993 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb6e60f5826562c15bcc HTTP/1.1" 200 None 2026-04-21 00:59:12,994 - lava_callback - DEBUG - Processing node 69e6cb6e60f5826562c15bcc: baseline-arm64 2026-04-21 00:59:13,269 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:13,365 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:13,366 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb6e60f5826562c15bcc/log.txt.gz 2026-04-21 00:59:13,510 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:13,649 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:13,651 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb6e60f5826562c15bcc/lava-logs.yaml 2026-04-21 00:59:13,657 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:13,751 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:13,753 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb6e60f5826562c15bcc/lava_callback.json.gz 2026-04-21 00:59:13,775 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:13,791 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb6e60f5826562c15bcc HTTP/1.1" 200 None 2026-04-21 00:59:13,792 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:13,808 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 00:59:13,809 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:13,830 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb6e60f5826562c15bcc HTTP/1.1" 200 None 2026-04-21 00:59:13,832 - lava_callback - INFO - Completed processing callback for node 69e6cb6e60f5826562c15bcc 2026-04-21 00:59:14,609 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:14,625 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 15 INFO: 172.20.0.1:50408 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50420 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51978 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:59:20,446 - lava_callback - INFO - Received callback for node 69e6cb7c60f5826562c15bf8 INFO: 172.20.0.1:51994 - "POST /node/69e6cb7c60f5826562c15bf8 HTTP/1.0" 202 Accepted 2026-04-21 00:59:20,452 - lava_callback - INFO - Starting async job submit for node 69e6cb7c60f5826562c15bf8 2026-04-21 00:59:20,474 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:20,487 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7c60f5826562c15bf8 HTTP/1.1" 200 None 2026-04-21 00:59:20,488 - lava_callback - DEBUG - Processing node 69e6cb7c60f5826562c15bf8: baseline-arm64 2026-04-21 00:59:20,714 - lava_callback - INFO - Received callback for node 69e6cb7a60f5826562c15bf1 INFO: 172.20.0.1:51996 - "POST /node/69e6cb7a60f5826562c15bf1 HTTP/1.0" 202 Accepted 2026-04-21 00:59:20,737 - lava_callback - INFO - Starting async job submit for node 69e6cb7a60f5826562c15bf1 2026-04-21 00:59:20,805 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:20,826 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:20,839 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7a60f5826562c15bf1 HTTP/1.1" 200 None 2026-04-21 00:59:20,840 - lava_callback - DEBUG - Processing node 69e6cb7a60f5826562c15bf1: baseline-arm64 2026-04-21 00:59:20,954 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:21,001 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7c60f5826562c15bf8/log.txt.gz 2026-04-21 00:59:21,390 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:21,399 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:21,503 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:21,505 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7a60f5826562c15bf1/log.txt.gz 2026-04-21 00:59:21,570 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:21,606 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7c60f5826562c15bf8/lava-logs.yaml 2026-04-21 00:59:21,686 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:21,686 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:21,812 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:21,813 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7c60f5826562c15bf8/lava_callback.json.gz 2026-04-21 00:59:21,836 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:21,851 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7c60f5826562c15bf8 HTTP/1.1" 200 None 2026-04-21 00:59:21,853 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:21,870 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:21,873 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7a60f5826562c15bf1/lava-logs.yaml 2026-04-21 00:59:21,875 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 00:59:21,878 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:21,881 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:21,900 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb7c60f5826562c15bf8 HTTP/1.1" 200 None 2026-04-21 00:59:21,903 - lava_callback - INFO - Completed processing callback for node 69e6cb7c60f5826562c15bf8 2026-04-21 00:59:22,004 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:22,005 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7a60f5826562c15bf1/lava_callback.json.gz 2026-04-21 00:59:22,033 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:22,049 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7a60f5826562c15bf1 HTTP/1.1" 200 None 2026-04-21 00:59:22,051 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:22,075 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 00:59:22,077 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:22,098 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb7a60f5826562c15bf1 HTTP/1.1" 200 None 2026-04-21 00:59:22,100 - lava_callback - INFO - Completed processing callback for node 69e6cb7a60f5826562c15bf1 2026-04-21 00:59:24,813 - lava_callback - INFO - Received callback for node 69e6cb8960f5826562c15c42 INFO: 172.20.0.1:52002 - "POST /node/69e6cb8960f5826562c15c42 HTTP/1.0" 202 Accepted 2026-04-21 00:59:24,824 - lava_callback - INFO - Starting async job submit for node 69e6cb8960f5826562c15c42 2026-04-21 00:59:24,851 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:24,865 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb8960f5826562c15c42 HTTP/1.1" 200 None 2026-04-21 00:59:24,865 - lava_callback - DEBUG - Processing node 69e6cb8960f5826562c15c42: baseline-nfs-arm64 2026-04-21 00:59:25,189 - lava_callback - INFO - Received callback for node 69e6cbb360f5826562c15cce INFO: 172.20.0.1:52008 - "POST /node/69e6cbb360f5826562c15cce HTTP/1.0" 202 Accepted 2026-04-21 00:59:25,213 - lava_callback - INFO - Starting async job submit for node 69e6cbb360f5826562c15cce 2026-04-21 00:59:25,293 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:25,365 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:25,365 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cbb360f5826562c15cce HTTP/1.1" 200 None 2026-04-21 00:59:25,366 - lava_callback - DEBUG - Processing node 69e6cbb360f5826562c15cce: baseline-arm64-foundriesio 2026-04-21 00:59:25,529 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:25,578 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb8960f5826562c15c42/log.txt.gz 2026-04-21 00:59:25,770 - lava_callback - INFO - Received callback for node 69e428dc3551964617338c75 2026-04-21 00:59:25,782 - lava_callback - INFO - Starting async job submit for node 69e428dc3551964617338c75 INFO: 172.20.0.1:52022 - "POST /node/69e428dc3551964617338c75 HTTP/1.0" 202 Accepted 2026-04-21 00:59:25,792 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:26,136 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e428dc3551964617338c75 HTTP/1.1" 200 None 2026-04-21 00:59:26,159 - lava_callback - DEBUG - Processing node 69e428dc3551964617338c75: baseline-arm-baylibre 2026-04-21 00:59:26,160 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e428dc3551964617338c75 2026-04-21 00:59:26,171 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:26,193 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:26,193 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:26,204 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e428dc3551964617338c75 HTTP/1.1" 200 None 2026-04-21 00:59:26,206 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:26,219 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc36005dbfa74e32f9687c HTTP/1.1" 200 None 2026-04-21 00:59:26,221 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:26,250 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e428dc3551964617338c75 HTTP/1.1" 200 None 2026-04-21 00:59:26,252 - lava_callback - ERROR - Error processing callback for node 69e428dc3551964617338c75: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-04-21 00:59:26,367 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:26,369 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-foundriesio-69e6cbb360f5826562c15cce/log.txt.gz 2026-04-21 00:59:26,420 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:26,462 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb8960f5826562c15c42/lava-logs.yaml INFO: 172.20.0.1:52028 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:59:26,552 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:26,650 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:26,771 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:26,773 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb8960f5826562c15c42/lava_callback.json.gz 2026-04-21 00:59:26,801 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:26,816 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb8960f5826562c15c42 HTTP/1.1" 200 None 2026-04-21 00:59:26,818 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:26,837 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 00:59:26,837 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:26,840 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:26,841 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-foundriesio-69e6cbb360f5826562c15cce/lava-logs.yaml 2026-04-21 00:59:26,852 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:26,863 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb8960f5826562c15c42 HTTP/1.1" 200 None 2026-04-21 00:59:26,865 - lava_callback - INFO - Completed processing callback for node 69e6cb8960f5826562c15c42 2026-04-21 00:59:27,018 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:27,020 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-foundriesio-69e6cbb360f5826562c15cce/lava_callback.json.gz 2026-04-21 00:59:27,066 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:27,090 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cbb360f5826562c15cce HTTP/1.1" 200 None 2026-04-21 00:59:27,092 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:27,110 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 00:59:27,112 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:27,144 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cbb360f5826562c15cce HTTP/1.1" 200 None 2026-04-21 00:59:27,146 - lava_callback - INFO - Completed processing callback for node 69e6cbb360f5826562c15cce 2026-04-21 00:59:29,456 - lava_callback - INFO - Received callback for node 69e6cbb260f5826562c15ccb INFO: 172.20.0.1:52032 - "POST /node/69e6cbb260f5826562c15ccb HTTP/1.0" 202 Accepted 2026-04-21 00:59:29,462 - lava_callback - INFO - Starting async job submit for node 69e6cbb260f5826562c15ccb 2026-04-21 00:59:29,483 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:29,497 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cbb260f5826562c15ccb HTTP/1.1" 200 None 2026-04-21 00:59:29,498 - lava_callback - DEBUG - Processing node 69e6cbb260f5826562c15ccb: baseline-arm64-foundriesio 2026-04-21 00:59:29,727 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:29,796 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:29,797 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-foundriesio-69e6cbb260f5826562c15ccb/log.txt.gz 2026-04-21 00:59:29,826 - lava_callback - INFO - Received callback for node 69e6cbae60f5826562c15cc2 INFO: 172.20.0.1:36970 - "POST /node/69e6cbae60f5826562c15cc2 HTTP/1.0" 202 Accepted 2026-04-21 00:59:29,842 - lava_callback - INFO - Starting async job submit for node 69e6cbae60f5826562c15cc2 2026-04-21 00:59:29,916 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:29,977 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cbae60f5826562c15cc2 HTTP/1.1" 200 None 2026-04-21 00:59:29,978 - lava_callback - DEBUG - Processing node 69e6cbae60f5826562c15cc2: baseline-arm64-pengutronix 2026-04-21 00:59:29,994 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:30,218 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:30,224 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-foundriesio-69e6cbb260f5826562c15ccb/lava-logs.yaml 2026-04-21 00:59:30,226 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:30,230 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:30,307 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:30,309 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69e6cbae60f5826562c15cc2/log.txt.gz 2026-04-21 00:59:30,328 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:30,365 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-foundriesio-69e6cbb260f5826562c15ccb/lava_callback.json.gz 2026-04-21 00:59:30,434 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:30,447 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:30,459 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cbb260f5826562c15ccb HTTP/1.1" 200 None 2026-04-21 00:59:30,461 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:30,475 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 00:59:30,476 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:30,499 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cbb260f5826562c15ccb HTTP/1.1" 200 None 2026-04-21 00:59:30,500 - lava_callback - INFO - Completed processing callback for node 69e6cbb260f5826562c15ccb 2026-04-21 00:59:30,623 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:30,626 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69e6cbae60f5826562c15cc2/lava-logs.yaml 2026-04-21 00:59:30,632 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:30,732 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:30,734 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69e6cbae60f5826562c15cc2/lava_callback.json.gz 2026-04-21 00:59:30,754 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:30,768 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cbae60f5826562c15cc2 HTTP/1.1" 200 None 2026-04-21 00:59:30,769 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:30,781 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 00:59:30,783 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:30,811 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cbae60f5826562c15cc2 HTTP/1.1" 200 None 2026-04-21 00:59:30,812 - lava_callback - INFO - Completed processing callback for node 69e6cbae60f5826562c15cc2 INFO: 172.20.0.1:36986 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:59:39,902 - lava_callback - INFO - Received callback for node 69e6c5d160f5826562c158bf INFO: 172.20.0.1:49216 - "POST /node/69e6c5d160f5826562c158bf HTTP/1.0" 202 Accepted 2026-04-21 00:59:39,913 - lava_callback - INFO - Starting async job submit for node 69e6c5d160f5826562c158bf 2026-04-21 00:59:39,949 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:39,963 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5d160f5826562c158bf HTTP/1.1" 200 None 2026-04-21 00:59:39,963 - lava_callback - DEBUG - Processing node 69e6c5d160f5826562c158bf: ltp-ipc 2026-04-21 00:59:40,043 - lava_callback - INFO - Received callback for node 69e6cb8160f5826562c15c04 INFO: 172.20.0.1:49224 - "POST /node/69e6cb8160f5826562c15c04 HTTP/1.0" 202 Accepted 2026-04-21 00:59:40,063 - lava_callback - INFO - Starting async job submit for node 69e6cb8160f5826562c15c04 2026-04-21 00:59:40,151 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:40,234 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb8160f5826562c15c04 HTTP/1.1" 200 None 2026-04-21 00:59:40,246 - lava_callback - DEBUG - Processing node 69e6cb8160f5826562c15c04: baseline-nfs-arm64 2026-04-21 00:59:40,700 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:40,729 - lava_callback - INFO - Received callback for node 69e6cb8a60f5826562c15c49 INFO: 172.20.0.1:49228 - "POST /node/69e6cb8a60f5826562c15c49 HTTP/1.0" 202 Accepted 2026-04-21 00:59:40,752 - lava_callback - INFO - Starting async job submit for node 69e6cb8a60f5826562c15c49 2026-04-21 00:59:40,816 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:40,849 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:40,859 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb8a60f5826562c15c49 HTTP/1.1" 200 None 2026-04-21 00:59:40,860 - lava_callback - DEBUG - Processing node 69e6cb8a60f5826562c15c49: baseline-nfs-arm64 2026-04-21 00:59:40,918 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:40,920 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-ipc-69e6c5d160f5826562c158bf/log.txt.gz 2026-04-21 00:59:41,133 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:41,241 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb8160f5826562c15c04/log.txt.gz 2026-04-21 00:59:41,697 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:41,804 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:41,825 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:41,932 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:41,935 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb8a60f5826562c15c49/log.txt.gz 2026-04-21 00:59:41,978 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:41,989 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:42,026 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-ipc-69e6c5d160f5826562c158bf/lava-logs.yaml 2026-04-21 00:59:42,032 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb8160f5826562c15c04/lava-logs.yaml 2026-04-21 00:59:42,137 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:42,154 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:42,192 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:42,293 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:42,294 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-ipc-69e6c5d160f5826562c158bf/lava_callback.json.gz 2026-04-21 00:59:42,321 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:42,336 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:42,336 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb8160f5826562c15c04/lava_callback.json.gz 2026-04-21 00:59:42,348 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:42,362 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:42,363 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb8a60f5826562c15c49/lava-logs.yaml 2026-04-21 00:59:42,372 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:42,378 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5d160f5826562c158bf HTTP/1.1" 200 None 2026-04-21 00:59:42,380 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb8160f5826562c15c04 HTTP/1.1" 200 None 2026-04-21 00:59:42,381 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:42,382 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:42,401 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 00:59:42,404 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 00:59:42,404 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:42,406 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:42,438 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c5d160f5826562c158bf HTTP/1.1" 200 None 2026-04-21 00:59:42,445 - lava_callback - INFO - Completed processing callback for node 69e6c5d160f5826562c158bf 2026-04-21 00:59:42,466 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb8160f5826562c15c04 HTTP/1.1" 200 None 2026-04-21 00:59:42,468 - lava_callback - INFO - Completed processing callback for node 69e6cb8160f5826562c15c04 2026-04-21 00:59:42,566 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:42,568 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb8a60f5826562c15c49/lava_callback.json.gz 2026-04-21 00:59:42,604 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:42,624 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb8a60f5826562c15c49 HTTP/1.1" 200 None 2026-04-21 00:59:42,626 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:42,640 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 00:59:42,642 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:42,670 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb8a60f5826562c15c49 HTTP/1.1" 200 None 2026-04-21 00:59:42,672 - lava_callback - INFO - Completed processing callback for node 69e6cb8a60f5826562c15c49 2026-04-21 00:59:43,498 - lava_callback - INFO - Received callback for node 69e6cbb060f5826562c15cc5 INFO: 172.20.0.1:49238 - "POST /node/69e6cbb060f5826562c15cc5 HTTP/1.0" 202 Accepted 2026-04-21 00:59:43,507 - lava_callback - INFO - Starting async job submit for node 69e6cbb060f5826562c15cc5 2026-04-21 00:59:43,528 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:43,542 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cbb060f5826562c15cc5 HTTP/1.1" 200 None 2026-04-21 00:59:43,543 - lava_callback - DEBUG - Processing node 69e6cbb060f5826562c15cc5: baseline-arm64-pengutronix 2026-04-21 00:59:43,751 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:43,823 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:43,825 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69e6cbb060f5826562c15cc5/log.txt.gz 2026-04-21 00:59:43,948 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:44,068 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:44,070 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69e6cbb060f5826562c15cc5/lava-logs.yaml 2026-04-21 00:59:44,076 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:44,165 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:44,167 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69e6cbb060f5826562c15cc5/lava_callback.json.gz 2026-04-21 00:59:44,187 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:44,206 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cbb060f5826562c15cc5 HTTP/1.1" 200 None 2026-04-21 00:59:44,207 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:44,220 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 00:59:44,221 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:44,241 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cbb060f5826562c15cc5 HTTP/1.1" 200 None 2026-04-21 00:59:44,243 - lava_callback - INFO - Completed processing callback for node 69e6cbb060f5826562c15cc5 2026-04-21 00:59:44,466 - lava_callback - INFO - Received callback for node 69e6cb9f60f5826562c15c77 INFO: 172.20.0.1:49240 - "POST /node/69e6cb9f60f5826562c15c77 HTTP/1.0" 202 Accepted 2026-04-21 00:59:44,472 - lava_callback - INFO - Starting async job submit for node 69e6cb9f60f5826562c15c77 2026-04-21 00:59:44,505 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:44,521 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb9f60f5826562c15c77 HTTP/1.1" 200 None 2026-04-21 00:59:44,522 - lava_callback - DEBUG - Processing node 69e6cb9f60f5826562c15c77: baseline-arm64-broonie 2026-04-21 00:59:44,638 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:44,720 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 15 2026-04-21 00:59:44,805 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:44,881 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:44,882 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cb9f60f5826562c15c77/log.txt.gz 2026-04-21 00:59:45,028 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:45,158 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:45,160 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cb9f60f5826562c15c77/lava-logs.yaml 2026-04-21 00:59:45,167 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:45,266 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:45,268 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cb9f60f5826562c15c77/lava_callback.json.gz 2026-04-21 00:59:45,290 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:45,304 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb9f60f5826562c15c77 HTTP/1.1" 200 None 2026-04-21 00:59:45,306 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:45,318 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 00:59:45,320 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:45,340 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb9f60f5826562c15c77 HTTP/1.1" 200 None 2026-04-21 00:59:45,341 - lava_callback - INFO - Completed processing callback for node 69e6cb9f60f5826562c15c77 INFO: 172.20.0.1:49242 - "GET / HTTP/1.0" 200 OK 2026-04-21 00:59:49,233 - lava_callback - INFO - Received callback for node 69e6cbad60f5826562c15cbf INFO: 172.20.0.1:49254 - "POST /node/69e6cbad60f5826562c15cbf HTTP/1.0" 202 Accepted 2026-04-21 00:59:49,239 - lava_callback - INFO - Starting async job submit for node 69e6cbad60f5826562c15cbf 2026-04-21 00:59:49,269 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:49,284 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cbad60f5826562c15cbf HTTP/1.1" 200 None 2026-04-21 00:59:49,284 - lava_callback - DEBUG - Processing node 69e6cbad60f5826562c15cbf: baseline-arm64-kontron 2026-04-21 00:59:49,568 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:49,651 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:49,653 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-kontron-69e6cbad60f5826562c15cbf/log.txt.gz 2026-04-21 00:59:49,829 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:49,964 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:49,966 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-kontron-69e6cbad60f5826562c15cbf/lava-logs.yaml 2026-04-21 00:59:49,974 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:50,081 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:50,082 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-kontron-69e6cbad60f5826562c15cbf/lava_callback.json.gz 2026-04-21 00:59:50,104 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:50,118 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cbad60f5826562c15cbf HTTP/1.1" 200 None 2026-04-21 00:59:50,120 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:50,133 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 00:59:50,134 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:50,161 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cbad60f5826562c15cbf HTTP/1.1" 200 None 2026-04-21 00:59:50,162 - lava_callback - INFO - Completed processing callback for node 69e6cbad60f5826562c15cbf 2026-04-21 00:59:51,647 - lava_callback - INFO - Received callback for node 69e6cb7e60f5826562c15bfe INFO: 172.20.0.1:49698 - "POST /node/69e6cb7e60f5826562c15bfe HTTP/1.0" 202 Accepted 2026-04-21 00:59:51,653 - lava_callback - INFO - Starting async job submit for node 69e6cb7e60f5826562c15bfe 2026-04-21 00:59:51,679 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:51,697 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7e60f5826562c15bfe HTTP/1.1" 200 None 2026-04-21 00:59:51,698 - lava_callback - DEBUG - Processing node 69e6cb7e60f5826562c15bfe: baseline-nfs-arm64 2026-04-21 00:59:52,073 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:52,160 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:52,162 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb7e60f5826562c15bfe/log.txt.gz 2026-04-21 00:59:52,366 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:52,501 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:52,503 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb7e60f5826562c15bfe/lava-logs.yaml 2026-04-21 00:59:52,511 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:52,618 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:52,619 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb7e60f5826562c15bfe/lava_callback.json.gz 2026-04-21 00:59:52,642 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:52,659 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7e60f5826562c15bfe HTTP/1.1" 200 None 2026-04-21 00:59:52,661 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:52,679 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 00:59:52,681 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:52,702 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb7e60f5826562c15bfe HTTP/1.1" 200 None 2026-04-21 00:59:52,703 - lava_callback - INFO - Completed processing callback for node 69e6cb7e60f5826562c15bfe 2026-04-21 00:59:52,860 - lava_callback - INFO - Received callback for node 69e6cbb160f5826562c15cc8 INFO: 172.20.0.1:49708 - "POST /node/69e6cbb160f5826562c15cc8 HTTP/1.0" 202 Accepted 2026-04-21 00:59:52,867 - lava_callback - INFO - Starting async job submit for node 69e6cbb160f5826562c15cc8 2026-04-21 00:59:52,890 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:52,904 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cbb160f5826562c15cc8 HTTP/1.1" 200 None 2026-04-21 00:59:52,905 - lava_callback - DEBUG - Processing node 69e6cbb160f5826562c15cc8: baseline-arm64-pengutronix 2026-04-21 00:59:53,160 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:53,181 - lava_callback - INFO - Received callback for node 69e429043551964617338c82 INFO: 172.20.0.1:49720 - "POST /node/69e429043551964617338c82 HTTP/1.0" 202 Accepted 2026-04-21 00:59:53,191 - lava_callback - INFO - Starting async job submit for node 69e429043551964617338c82 2026-04-21 00:59:53,192 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:53,210 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e429043551964617338c82 HTTP/1.1" 200 None 2026-04-21 00:59:53,211 - lava_callback - DEBUG - Processing node 69e429043551964617338c82: baseline-arm-clabbe 2026-04-21 00:59:53,211 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e429043551964617338c82 2026-04-21 00:59:53,212 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:53,238 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e429043551964617338c82 HTTP/1.1" 200 None 2026-04-21 00:59:53,239 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:53,249 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:53,251 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69e6cbb160f5826562c15cc8/log.txt.gz 2026-04-21 00:59:53,296 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc36005dbfa74e32f9687c HTTP/1.1" 200 None 2026-04-21 00:59:53,315 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:53,433 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:53,434 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e429043551964617338c82 HTTP/1.1" 200 None 2026-04-21 00:59:53,437 - lava_callback - ERROR - Error processing callback for node 69e429043551964617338c82: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-04-21 00:59:53,567 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:53,569 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69e6cbb160f5826562c15cc8/lava-logs.yaml 2026-04-21 00:59:53,576 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 00:59:53,662 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 00:59:53,663 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69e6cbb160f5826562c15cc8/lava_callback.json.gz 2026-04-21 00:59:53,684 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:53,706 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cbb160f5826562c15cc8 HTTP/1.1" 200 None 2026-04-21 00:59:53,707 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:53,720 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 00:59:53,722 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 00:59:53,758 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cbb160f5826562c15cc8 HTTP/1.1" 200 None 2026-04-21 00:59:53,759 - lava_callback - INFO - Completed processing callback for node 69e6cbb160f5826562c15cc8 INFO: 172.20.0.1:49736 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:00:00,092 - lava_callback - INFO - Received callback for node 69e6c5d460f5826562c158c5 INFO: 172.20.0.1:41132 - "POST /node/69e6c5d460f5826562c158c5 HTTP/1.0" 202 Accepted 2026-04-21 01:00:00,102 - lava_callback - INFO - Starting async job submit for node 69e6c5d460f5826562c158c5 2026-04-21 01:00:00,138 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:00,154 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5d460f5826562c158c5 HTTP/1.1" 200 None 2026-04-21 01:00:00,155 - lava_callback - DEBUG - Processing node 69e6c5d460f5826562c158c5: ltp-sched 2026-04-21 01:00:00,600 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:00:00,691 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:00:00,693 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-sched-69e6c5d460f5826562c158c5/log.txt.gz 2026-04-21 01:00:00,896 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:00:01,089 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:00:01,092 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-sched-69e6c5d460f5826562c158c5/lava-logs.yaml 2026-04-21 01:00:01,100 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:00:01,203 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:00:01,205 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-sched-69e6c5d460f5826562c158c5/lava_callback.json.gz 2026-04-21 01:00:01,233 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:01,247 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5d460f5826562c158c5 HTTP/1.1" 200 None 2026-04-21 01:00:01,249 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:01,264 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 01:00:01,265 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:01,287 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c5d460f5826562c158c5 HTTP/1.1" 200 None 2026-04-21 01:00:01,288 - lava_callback - INFO - Completed processing callback for node 69e6c5d460f5826562c158c5 2026-04-21 01:00:01,515 - lava_callback - INFO - Received callback for node 69e6cb9c60f5826562c15c70 INFO: 172.20.0.1:41136 - "POST /node/69e6cb9c60f5826562c15c70 HTTP/1.0" 202 Accepted 2026-04-21 01:00:01,521 - lava_callback - INFO - Starting async job submit for node 69e6cb9c60f5826562c15c70 2026-04-21 01:00:01,544 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:01,563 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb9c60f5826562c15c70 HTTP/1.1" 200 None 2026-04-21 01:00:01,564 - lava_callback - DEBUG - Processing node 69e6cb9c60f5826562c15c70: baseline-arm64-broonie 2026-04-21 01:00:01,879 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:00:01,965 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:00:01,967 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cb9c60f5826562c15c70/log.txt.gz 2026-04-21 01:00:02,130 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:00:02,278 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:00:02,280 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cb9c60f5826562c15c70/lava-logs.yaml 2026-04-21 01:00:02,287 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:00:02,380 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:00:02,382 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cb9c60f5826562c15c70/lava_callback.json.gz 2026-04-21 01:00:02,405 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:02,419 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb9c60f5826562c15c70 HTTP/1.1" 200 None 2026-04-21 01:00:02,421 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:02,433 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 01:00:02,435 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:02,461 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb9c60f5826562c15c70 HTTP/1.1" 200 None 2026-04-21 01:00:02,463 - lava_callback - INFO - Completed processing callback for node 69e6cb9c60f5826562c15c70 2026-04-21 01:00:02,485 - lava_callback - INFO - Received callback for node 69e6c5d560f5826562c158c9 INFO: 172.20.0.1:41152 - "POST /node/69e6c5d560f5826562c158c9 HTTP/1.0" 202 Accepted 2026-04-21 01:00:02,511 - lava_callback - INFO - Starting async job submit for node 69e6c5d560f5826562c158c9 2026-04-21 01:00:02,561 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:02,575 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5d560f5826562c158c9 HTTP/1.1" 200 None 2026-04-21 01:00:02,576 - lava_callback - DEBUG - Processing node 69e6c5d560f5826562c158c9: ltp-smoke 2026-04-21 01:00:02,961 - lava_callback - INFO - Received callback for node 69e6cba360f5826562c15c7f INFO: 172.20.0.1:41162 - "POST /node/69e6cba360f5826562c15c7f HTTP/1.0" 202 Accepted 2026-04-21 01:00:02,975 - lava_callback - INFO - Starting async job submit for node 69e6cba360f5826562c15c7f 2026-04-21 01:00:02,999 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:02,999 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:00:03,026 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cba360f5826562c15c7f HTTP/1.1" 200 None 2026-04-21 01:00:03,027 - lava_callback - DEBUG - Processing node 69e6cba360f5826562c15c7f: baseline-arm64-broonie 2026-04-21 01:00:03,095 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:00:03,138 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-smoke-69e6c5d560f5826562c158c9/log.txt.gz 2026-04-21 01:00:03,486 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:00:03,513 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:00:03,602 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:00:03,604 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cba360f5826562c15c7f/log.txt.gz 2026-04-21 01:00:03,672 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:00:03,701 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-smoke-69e6c5d560f5826562c158c9/lava-logs.yaml 2026-04-21 01:00:03,765 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:00:03,765 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:00:03,872 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:00:03,874 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-smoke-69e6c5d560f5826562c158c9/lava_callback.json.gz 2026-04-21 01:00:03,901 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:03,914 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:00:03,916 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cba360f5826562c15c7f/lava-logs.yaml 2026-04-21 01:00:03,916 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5d560f5826562c158c9 HTTP/1.1" 200 None 2026-04-21 01:00:03,918 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:03,923 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:00:03,935 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 01:00:03,937 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:03,968 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c5d560f5826562c158c9 HTTP/1.1" 200 None 2026-04-21 01:00:03,970 - lava_callback - INFO - Completed processing callback for node 69e6c5d560f5826562c158c9 2026-04-21 01:00:04,056 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:00:04,058 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cba360f5826562c15c7f/lava_callback.json.gz 2026-04-21 01:00:04,093 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:04,111 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cba360f5826562c15c7f HTTP/1.1" 200 None 2026-04-21 01:00:04,113 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:04,132 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 01:00:04,134 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:04,163 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cba360f5826562c15c7f HTTP/1.1" 200 None 2026-04-21 01:00:04,164 - lava_callback - INFO - Completed processing callback for node 69e6cba360f5826562c15c7f INFO: 172.20.0.1:41176 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:00:14,728 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:14,748 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 15 INFO: 172.20.0.1:51502 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51514 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49182 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:00:20,688 - lava_callback - INFO - Received callback for node 69e6cba160f5826562c15c7a INFO: 172.20.0.1:49186 - "POST /node/69e6cba160f5826562c15c7a HTTP/1.0" 202 Accepted 2026-04-21 01:00:20,694 - lava_callback - INFO - Starting async job submit for node 69e6cba160f5826562c15c7a 2026-04-21 01:00:20,717 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:20,730 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cba160f5826562c15c7a HTTP/1.1" 200 None 2026-04-21 01:00:20,731 - lava_callback - DEBUG - Processing node 69e6cba160f5826562c15c7a: baseline-arm64-broonie 2026-04-21 01:00:21,037 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:00:21,113 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:00:21,115 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cba160f5826562c15c7a/log.txt.gz 2026-04-21 01:00:21,275 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:00:21,406 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:00:21,408 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cba160f5826562c15c7a/lava-logs.yaml 2026-04-21 01:00:21,416 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:00:21,505 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:00:21,507 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cba160f5826562c15c7a/lava_callback.json.gz 2026-04-21 01:00:21,531 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:21,546 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cba160f5826562c15c7a HTTP/1.1" 200 None 2026-04-21 01:00:21,547 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:21,568 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 01:00:21,570 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:21,590 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cba160f5826562c15c7a HTTP/1.1" 200 None 2026-04-21 01:00:21,592 - lava_callback - INFO - Completed processing callback for node 69e6cba160f5826562c15c7a 2026-04-21 01:00:22,721 - lava_callback - INFO - Received callback for node 69e6cba460f5826562c15c82 INFO: 172.20.0.1:49202 - "POST /node/69e6cba460f5826562c15c82 HTTP/1.0" 202 Accepted 2026-04-21 01:00:22,727 - lava_callback - INFO - Starting async job submit for node 69e6cba460f5826562c15c82 2026-04-21 01:00:22,750 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:22,763 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cba460f5826562c15c82 HTTP/1.1" 200 None 2026-04-21 01:00:22,764 - lava_callback - DEBUG - Processing node 69e6cba460f5826562c15c82: baseline-arm64-broonie 2026-04-21 01:00:23,033 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:00:23,164 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:00:23,166 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cba460f5826562c15c82/log.txt.gz 2026-04-21 01:00:23,328 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:00:23,500 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:00:23,503 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cba460f5826562c15c82/lava-logs.yaml 2026-04-21 01:00:23,512 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:00:23,658 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:00:23,660 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cba460f5826562c15c82/lava_callback.json.gz 2026-04-21 01:00:23,683 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:23,696 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cba460f5826562c15c82 HTTP/1.1" 200 None 2026-04-21 01:00:23,698 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:23,711 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 01:00:23,712 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:23,733 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cba460f5826562c15c82 HTTP/1.1" 200 None 2026-04-21 01:00:23,735 - lava_callback - INFO - Completed processing callback for node 69e6cba460f5826562c15c82 INFO: 172.20.0.1:49208 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:00:33,202 - lava_callback - INFO - Received callback for node 69e4292e3551964617338c8f INFO: 172.20.0.1:34828 - "POST /node/69e4292e3551964617338c8f HTTP/1.0" 202 Accepted 2026-04-21 01:00:33,208 - lava_callback - INFO - Starting async job submit for node 69e4292e3551964617338c8f 2026-04-21 01:00:33,209 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:33,222 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4292e3551964617338c8f HTTP/1.1" 200 None 2026-04-21 01:00:33,223 - lava_callback - DEBUG - Processing node 69e4292e3551964617338c8f: baseline-arm-clabbe 2026-04-21 01:00:33,223 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4292e3551964617338c8f 2026-04-21 01:00:33,224 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:33,237 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4292e3551964617338c8f HTTP/1.1" 200 None 2026-04-21 01:00:33,238 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:33,251 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc36005dbfa74e32f9687c HTTP/1.1" 200 None 2026-04-21 01:00:33,252 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:33,270 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4292e3551964617338c8f HTTP/1.1" 200 None 2026-04-21 01:00:33,271 - lava_callback - ERROR - Error processing callback for node 69e4292e3551964617338c8f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:34844 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:00:40,017 - lava_callback - INFO - Received callback for node 69e6c5d860f5826562c158cf INFO: 172.20.0.1:53182 - "POST /node/69e6c5d860f5826562c158cf HTTP/1.0" 202 Accepted 2026-04-21 01:00:40,023 - lava_callback - INFO - Starting async job submit for node 69e6c5d860f5826562c158cf 2026-04-21 01:00:40,049 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:40,062 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5d860f5826562c158cf HTTP/1.1" 200 None 2026-04-21 01:00:40,063 - lava_callback - DEBUG - Processing node 69e6c5d860f5826562c158cf: ltp-syscalls-ipc 2026-04-21 01:00:40,428 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:00:40,692 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:00:40,694 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-syscalls-ipc-69e6c5d860f5826562c158cf/log.txt.gz 2026-04-21 01:00:40,895 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:00:41,067 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:00:41,069 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-syscalls-ipc-69e6c5d860f5826562c158cf/lava-logs.yaml 2026-04-21 01:00:41,078 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:00:41,204 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:00:41,205 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-syscalls-ipc-69e6c5d860f5826562c158cf/lava_callback.json.gz 2026-04-21 01:00:41,230 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:41,247 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5d860f5826562c158cf HTTP/1.1" 200 None 2026-04-21 01:00:41,248 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:41,262 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 01:00:41,263 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:41,285 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c5d860f5826562c158cf HTTP/1.1" 200 None 2026-04-21 01:00:41,286 - lava_callback - INFO - Completed processing callback for node 69e6c5d860f5826562c158cf 2026-04-21 01:00:44,750 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:00:44,766 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:53194 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43928 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56434 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:01:13,175 - lava_callback - INFO - Received callback for node 69e429543551964617338c9e INFO: 172.20.0.1:48808 - "POST /node/69e429543551964617338c9e HTTP/1.0" 202 Accepted 2026-04-21 01:01:13,183 - lava_callback - INFO - Starting async job submit for node 69e429543551964617338c9e 2026-04-21 01:01:13,184 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:01:13,205 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e429543551964617338c9e HTTP/1.1" 200 None 2026-04-21 01:01:13,206 - lava_callback - DEBUG - Processing node 69e429543551964617338c9e: baseline-arm-clabbe 2026-04-21 01:01:13,207 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e429543551964617338c9e 2026-04-21 01:01:13,208 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:01:13,259 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e429543551964617338c9e HTTP/1.1" 200 None 2026-04-21 01:01:13,262 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:01:13,291 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc36005dbfa74e32f9687c HTTP/1.1" 200 None 2026-04-21 01:01:13,294 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:01:13,320 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e429543551964617338c9e HTTP/1.1" 200 None 2026-04-21 01:01:13,321 - lava_callback - ERROR - Error processing callback for node 69e429543551964617338c9e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:48822 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48838 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43500 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43502 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39486 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38280 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:01:53,212 - lava_callback - INFO - Received callback for node 69e429793551964617338ca9 INFO: 172.20.0.1:46310 - "POST /node/69e429793551964617338ca9 HTTP/1.0" 202 Accepted 2026-04-21 01:01:53,218 - lava_callback - INFO - Starting async job submit for node 69e429793551964617338ca9 2026-04-21 01:01:53,219 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:01:53,233 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e429793551964617338ca9 HTTP/1.1" 200 None 2026-04-21 01:01:53,234 - lava_callback - DEBUG - Processing node 69e429793551964617338ca9: baseline-arm-clabbe 2026-04-21 01:01:53,234 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e429793551964617338ca9 2026-04-21 01:01:53,235 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:01:53,249 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e429793551964617338ca9 HTTP/1.1" 200 None 2026-04-21 01:01:53,250 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:01:53,265 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc36005dbfa74e32f9687c HTTP/1.1" 200 None 2026-04-21 01:01:53,267 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:01:53,290 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e429793551964617338ca9 HTTP/1.1" 200 None 2026-04-21 01:01:53,291 - lava_callback - ERROR - Error processing callback for node 69e429793551964617338ca9: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:46320 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43242 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43810 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43818 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33776 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:02:23,182 - lava_callback - INFO - Received callback for node 69e6cb9e60f5826562c15c74 INFO: 172.20.0.1:33790 - "POST /node/69e6cb9e60f5826562c15c74 HTTP/1.0" 202 Accepted 2026-04-21 01:02:23,189 - lava_callback - INFO - Starting async job submit for node 69e6cb9e60f5826562c15c74 2026-04-21 01:02:23,211 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:02:23,226 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb9e60f5826562c15c74 HTTP/1.1" 200 None 2026-04-21 01:02:23,226 - lava_callback - DEBUG - Processing node 69e6cb9e60f5826562c15c74: baseline-arm64-broonie 2026-04-21 01:02:23,530 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:02:23,644 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:02:23,646 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cb9e60f5826562c15c74/log.txt.gz 2026-04-21 01:02:23,828 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:02:24,018 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:02:24,020 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cb9e60f5826562c15c74/lava-logs.yaml 2026-04-21 01:02:24,028 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:02:24,150 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:02:24,152 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cb9e60f5826562c15c74/lava_callback.json.gz 2026-04-21 01:02:24,175 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:02:24,188 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb9e60f5826562c15c74 HTTP/1.1" 200 None 2026-04-21 01:02:24,190 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:02:24,207 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 01:02:24,209 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:02:24,229 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb9e60f5826562c15c74 HTTP/1.1" 200 None 2026-04-21 01:02:24,230 - lava_callback - INFO - Completed processing callback for node 69e6cb9e60f5826562c15c74 INFO: 172.20.0.1:33800 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:02:33,309 - lava_callback - INFO - Received callback for node 69e4299d3551964617338cb6 INFO: 172.20.0.1:45056 - "POST /node/69e4299d3551964617338cb6 HTTP/1.0" 202 Accepted 2026-04-21 01:02:33,315 - lava_callback - INFO - Starting async job submit for node 69e4299d3551964617338cb6 2026-04-21 01:02:33,316 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:02:33,330 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4299d3551964617338cb6 HTTP/1.1" 200 None 2026-04-21 01:02:33,331 - lava_callback - DEBUG - Processing node 69e4299d3551964617338cb6: baseline-arm-clabbe 2026-04-21 01:02:33,331 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4299d3551964617338cb6 2026-04-21 01:02:33,332 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:02:33,345 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4299d3551964617338cb6 HTTP/1.1" 200 None 2026-04-21 01:02:33,346 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:02:33,360 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc36005dbfa74e32f9687c HTTP/1.1" 200 None 2026-04-21 01:02:33,361 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:02:33,378 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4299d3551964617338cb6 HTTP/1.1" 200 None 2026-04-21 01:02:33,379 - lava_callback - ERROR - Error processing callback for node 69e4299d3551964617338cb6: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:45060 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:02:44,768 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:02:44,785 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:45532 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45636 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44142 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:03:13,285 - lava_callback - INFO - Received callback for node 69e429c33551964617338cc3 INFO: 172.20.0.1:40498 - "POST /node/69e429c33551964617338cc3 HTTP/1.0" 202 Accepted 2026-04-21 01:03:13,291 - lava_callback - INFO - Starting async job submit for node 69e429c33551964617338cc3 2026-04-21 01:03:13,292 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:03:13,313 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e429c33551964617338cc3 HTTP/1.1" 200 None 2026-04-21 01:03:13,315 - lava_callback - DEBUG - Processing node 69e429c33551964617338cc3: baseline-arm-clabbe 2026-04-21 01:03:13,315 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e429c33551964617338cc3 2026-04-21 01:03:13,316 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:03:13,348 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e429c33551964617338cc3 HTTP/1.1" 200 None 2026-04-21 01:03:13,350 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:03:13,366 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc36005dbfa74e32f9687c HTTP/1.1" 200 None 2026-04-21 01:03:13,367 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:03:13,388 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e429c33551964617338cc3 HTTP/1.1" 200 None 2026-04-21 01:03:13,389 - lava_callback - ERROR - Error processing callback for node 69e429c33551964617338cc3: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:40510 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40512 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45698 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45704 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46812 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:03:40,023 - lava_callback - INFO - Received callback for node 69e6c5da60f5826562c158d2 INFO: 172.20.0.1:58150 - "POST /node/69e6c5da60f5826562c158d2 HTTP/1.0" 202 Accepted 2026-04-21 01:03:40,029 - lava_callback - INFO - Starting async job submit for node 69e6c5da60f5826562c158d2 2026-04-21 01:03:40,055 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:03:40,069 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5da60f5826562c158d2 HTTP/1.1" 200 None 2026-04-21 01:03:40,070 - lava_callback - DEBUG - Processing node 69e6c5da60f5826562c158d2: ltp-timers 2026-04-21 01:03:40,430 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:03:40,543 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:03:40,544 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-timers-69e6c5da60f5826562c158d2/log.txt.gz 2026-04-21 01:03:40,852 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:03:41,036 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:03:41,038 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-timers-69e6c5da60f5826562c158d2/lava-logs.yaml 2026-04-21 01:03:41,049 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:03:41,339 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:03:41,341 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-timers-69e6c5da60f5826562c158d2/lava_callback.json.gz 2026-04-21 01:03:41,366 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:03:41,380 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5da60f5826562c158d2 HTTP/1.1" 200 None 2026-04-21 01:03:41,381 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:03:41,395 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 01:03:41,396 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:03:41,420 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c5da60f5826562c158d2 HTTP/1.1" 200 None 2026-04-21 01:03:41,421 - lava_callback - INFO - Completed processing callback for node 69e6c5da60f5826562c158d2 2026-04-21 01:03:44,787 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:03:44,806 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:58152 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:03:53,375 - lava_callback - INFO - Received callback for node 69e429ea3551964617338cd2 INFO: 172.20.0.1:40680 - "POST /node/69e429ea3551964617338cd2 HTTP/1.0" 202 Accepted 2026-04-21 01:03:53,380 - lava_callback - INFO - Starting async job submit for node 69e429ea3551964617338cd2 2026-04-21 01:03:53,381 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:03:53,396 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e429ea3551964617338cd2 HTTP/1.1" 200 None 2026-04-21 01:03:53,397 - lava_callback - DEBUG - Processing node 69e429ea3551964617338cd2: baseline-arm-clabbe 2026-04-21 01:03:53,397 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e429ea3551964617338cd2 2026-04-21 01:03:53,398 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:03:53,413 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e429ea3551964617338cd2 HTTP/1.1" 200 None 2026-04-21 01:03:53,415 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:03:53,430 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc36005dbfa74e32f9687c HTTP/1.1" 200 None 2026-04-21 01:03:53,431 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:03:53,461 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e429ea3551964617338cd2 HTTP/1.1" 200 None 2026-04-21 01:03:53,462 - lava_callback - ERROR - Error processing callback for node 69e429ea3551964617338cd2: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:40690 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:04:05,813 - lava_callback - INFO - Received callback for node 69e6c5db60f5826562c158d5 INFO: 172.20.0.1:52064 - "POST /node/69e6c5db60f5826562c158d5 HTTP/1.0" 202 Accepted 2026-04-21 01:04:05,819 - lava_callback - INFO - Starting async job submit for node 69e6c5db60f5826562c158d5 2026-04-21 01:04:05,844 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:04:05,858 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5db60f5826562c158d5 HTTP/1.1" 200 None 2026-04-21 01:04:05,859 - lava_callback - DEBUG - Processing node 69e6c5db60f5826562c158d5: ltp-watchqueue 2026-04-21 01:04:06,204 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:04:06,291 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:04:06,292 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-watchqueue-69e6c5db60f5826562c158d5/log.txt.gz 2026-04-21 01:04:06,514 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.20.0.1:52072 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:04:06,698 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:04:06,700 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-watchqueue-69e6c5db60f5826562c158d5/lava-logs.yaml 2026-04-21 01:04:06,709 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:04:06,832 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:04:06,834 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-watchqueue-69e6c5db60f5826562c158d5/lava_callback.json.gz 2026-04-21 01:04:06,867 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:04:06,884 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c5db60f5826562c158d5 HTTP/1.1" 200 None 2026-04-21 01:04:06,885 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:04:06,898 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 01:04:06,899 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:04:06,918 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6c5db60f5826562c158d5 HTTP/1.1" 200 None 2026-04-21 01:04:06,920 - lava_callback - INFO - Completed processing callback for node 69e6c5db60f5826562c158d5 2026-04-21 01:04:14,808 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:04:14,823 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:40070 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40072 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53162 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53170 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:04:33,325 - lava_callback - INFO - Received callback for node 69e42a133551964617338cdd INFO: 172.20.0.1:55452 - "POST /node/69e42a133551964617338cdd HTTP/1.0" 202 Accepted 2026-04-21 01:04:33,330 - lava_callback - INFO - Starting async job submit for node 69e42a133551964617338cdd 2026-04-21 01:04:33,331 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:04:33,347 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42a133551964617338cdd HTTP/1.1" 200 None 2026-04-21 01:04:33,348 - lava_callback - DEBUG - Processing node 69e42a133551964617338cdd: baseline-arm-clabbe 2026-04-21 01:04:33,348 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e42a133551964617338cdd 2026-04-21 01:04:33,348 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:04:33,362 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42a133551964617338cdd HTTP/1.1" 200 None 2026-04-21 01:04:33,363 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:04:33,376 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc36005dbfa74e32f9687c HTTP/1.1" 200 None 2026-04-21 01:04:33,378 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:04:33,395 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e42a133551964617338cdd HTTP/1.1" 200 None 2026-04-21 01:04:33,396 - lava_callback - ERROR - Error processing callback for node 69e42a133551964617338cdd: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:55454 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35766 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39618 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:05:02,807 - lava_callback - INFO - Received callback for node 69e6cb7d60f5826562c15bfb INFO: 172.20.0.1:38964 - "POST /node/69e6cb7d60f5826562c15bfb HTTP/1.0" 202 Accepted 2026-04-21 01:05:02,814 - lava_callback - INFO - Starting async job submit for node 69e6cb7d60f5826562c15bfb 2026-04-21 01:05:02,850 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:05:02,864 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7d60f5826562c15bfb HTTP/1.1" 200 None 2026-04-21 01:05:02,865 - lava_callback - DEBUG - Processing node 69e6cb7d60f5826562c15bfb: baseline-arm64 2026-04-21 01:05:03,235 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:05:03,347 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:05:03,348 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7d60f5826562c15bfb/log.txt.gz 2026-04-21 01:05:03,557 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:05:03,707 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:05:03,709 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7d60f5826562c15bfb/lava-logs.yaml 2026-04-21 01:05:03,718 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:05:03,838 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:05:03,840 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7d60f5826562c15bfb/lava_callback.json.gz DEBUG: Setup failed for 69e6cb7d60f5826562c15bfb. Transitting job node result: incomplete -> fail 2026-04-21 01:05:03,912 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:05:03,925 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7d60f5826562c15bfb HTTP/1.1" 200 None 2026-04-21 01:05:03,927 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:05:03,949 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 01:05:03,951 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:05:03,973 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb7d60f5826562c15bfb HTTP/1.1" 200 None 2026-04-21 01:05:04,047 - lava_callback - INFO - Completed processing callback for node 69e6cb7d60f5826562c15bfb 2026-04-21 01:05:06,122 - lava_callback - INFO - Received callback for node 69e42a3d3551964617338ced INFO: 172.20.0.1:38976 - "POST /node/69e42a3d3551964617338ced HTTP/1.0" 202 Accepted 2026-04-21 01:05:06,127 - lava_callback - INFO - Starting async job submit for node 69e42a3d3551964617338ced 2026-04-21 01:05:06,129 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:05:06,142 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42a3d3551964617338ced HTTP/1.1" 200 None 2026-04-21 01:05:06,143 - lava_callback - DEBUG - Processing node 69e42a3d3551964617338ced: baseline-arm64-baylibre 2026-04-21 01:05:06,143 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e42a3d3551964617338ced 2026-04-21 01:05:06,144 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:05:06,157 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42a3d3551964617338ced HTTP/1.1" 200 None 2026-04-21 01:05:06,158 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:05:06,171 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 01:05:06,172 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:05:06,191 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e42a3d3551964617338ced HTTP/1.1" 200 None 2026-04-21 01:05:06,192 - lava_callback - ERROR - Error processing callback for node 69e42a3d3551964617338ced: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:38978 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:05:14,825 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:05:14,841 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:58762 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58776 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44158 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:05:21,812 - lava_callback - INFO - Received callback for node 69e6cb7660f5826562c15be4 INFO: 172.20.0.1:44172 - "POST /node/69e6cb7660f5826562c15be4 HTTP/1.0" 202 Accepted 2026-04-21 01:05:21,819 - lava_callback - INFO - Starting async job submit for node 69e6cb7660f5826562c15be4 2026-04-21 01:05:21,852 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:05:21,866 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7660f5826562c15be4 HTTP/1.1" 200 None 2026-04-21 01:05:21,867 - lava_callback - DEBUG - Processing node 69e6cb7660f5826562c15be4: baseline-arm64 2026-04-21 01:05:22,568 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:05:22,694 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:05:22,696 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7660f5826562c15be4/log.txt.gz 2026-04-21 01:05:23,169 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:05:23,353 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:05:23,355 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7660f5826562c15be4/lava-logs.yaml 2026-04-21 01:05:23,370 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:05:23,474 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:05:23,476 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7660f5826562c15be4/lava_callback.json.gz 2026-04-21 01:05:23,508 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:05:23,522 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7660f5826562c15be4 HTTP/1.1" 200 None 2026-04-21 01:05:23,524 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:05:23,548 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 01:05:23,549 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:05:23,570 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb7660f5826562c15be4 HTTP/1.1" 200 None 2026-04-21 01:05:23,571 - lava_callback - INFO - Completed processing callback for node 69e6cb7660f5826562c15be4 INFO: 172.20.0.1:44186 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39664 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:05:44,843 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:05:44,859 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:53082 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55258 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58792 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:06:08,604 - lava_callback - INFO - Received callback for node 69e6cb8660f5826562c15c13 INFO: 172.20.0.1:58802 - "POST /node/69e6cb8660f5826562c15c13 HTTP/1.0" 202 Accepted 2026-04-21 01:06:08,611 - lava_callback - INFO - Starting async job submit for node 69e6cb8660f5826562c15c13 2026-04-21 01:06:08,643 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:06:08,657 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb8660f5826562c15c13 HTTP/1.1" 200 None 2026-04-21 01:06:08,658 - lava_callback - DEBUG - Processing node 69e6cb8660f5826562c15c13: baseline-nfs-arm64 2026-04-21 01:06:09,211 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:06:09,356 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:06:09,357 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb8660f5826562c15c13/log.txt.gz 2026-04-21 01:06:09,742 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:06:09,925 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:06:09,927 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb8660f5826562c15c13/lava-logs.yaml 2026-04-21 01:06:09,940 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:06:10,072 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:06:10,074 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb8660f5826562c15c13/lava_callback.json.gz 2026-04-21 01:06:10,106 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:06:10,121 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb8660f5826562c15c13 HTTP/1.1" 200 None 2026-04-21 01:06:10,122 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:06:10,136 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 01:06:10,138 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:06:10,166 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb8660f5826562c15c13 HTTP/1.1" 200 None 2026-04-21 01:06:10,168 - lava_callback - INFO - Completed processing callback for node 69e6cb8660f5826562c15c13 2026-04-21 01:06:14,861 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:06:14,883 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:33398 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33414 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38414 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38424 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:06:33,458 - lava_callback - INFO - Received callback for node 69e42a843551964617338d10 INFO: 172.20.0.1:34712 - "POST /node/69e42a843551964617338d10 HTTP/1.0" 202 Accepted 2026-04-21 01:06:33,464 - lava_callback - INFO - Starting async job submit for node 69e42a843551964617338d10 2026-04-21 01:06:33,465 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:06:33,479 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42a843551964617338d10 HTTP/1.1" 200 None 2026-04-21 01:06:33,480 - lava_callback - DEBUG - Processing node 69e42a843551964617338d10: baseline-arm-clabbe 2026-04-21 01:06:33,480 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e42a843551964617338d10 2026-04-21 01:06:33,481 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:06:33,496 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42a843551964617338d10 HTTP/1.1" 200 None 2026-04-21 01:06:33,498 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:06:33,533 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc36005dbfa74e32f9687c HTTP/1.1" 200 None 2026-04-21 01:06:33,535 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:06:33,559 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e42a843551964617338d10 HTTP/1.1" 200 None 2026-04-21 01:06:33,560 - lava_callback - ERROR - Error processing callback for node 69e42a843551964617338d10: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:34726 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56440 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50576 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:07:06,144 - lava_callback - INFO - Received callback for node 69e42aaf3551964617338d1d INFO: 172.20.0.1:43902 - "POST /node/69e42aaf3551964617338d1d HTTP/1.0" 202 Accepted 2026-04-21 01:07:06,150 - lava_callback - INFO - Starting async job submit for node 69e42aaf3551964617338d1d 2026-04-21 01:07:06,151 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:07:06,165 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42aaf3551964617338d1d HTTP/1.1" 200 None 2026-04-21 01:07:06,166 - lava_callback - DEBUG - Processing node 69e42aaf3551964617338d1d: baseline-arm64-baylibre 2026-04-21 01:07:06,166 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e42aaf3551964617338d1d 2026-04-21 01:07:06,166 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:07:06,179 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42aaf3551964617338d1d HTTP/1.1" 200 None 2026-04-21 01:07:06,180 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:07:06,193 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 01:07:06,194 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:07:06,211 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e42aaf3551964617338d1d HTTP/1.1" 200 None 2026-04-21 01:07:06,212 - lava_callback - ERROR - Error processing callback for node 69e42aaf3551964617338d1d: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:43916 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45150 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45152 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38060 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38064 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50182 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57808 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41982 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48462 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57730 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57738 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43144 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:08:26,256 - lava_callback - INFO - Received callback for node 69e42afa3551964617338d41 INFO: 172.20.0.1:43156 - "POST /node/69e42afa3551964617338d41 HTTP/1.0" 202 Accepted 2026-04-21 01:08:26,262 - lava_callback - INFO - Starting async job submit for node 69e42afa3551964617338d41 2026-04-21 01:08:26,263 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:08:26,277 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42afa3551964617338d41 HTTP/1.1" 200 None 2026-04-21 01:08:26,278 - lava_callback - DEBUG - Processing node 69e42afa3551964617338d41: baseline-arm64-baylibre 2026-04-21 01:08:26,278 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e42afa3551964617338d41 2026-04-21 01:08:26,279 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:08:26,292 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42afa3551964617338d41 HTTP/1.1" 200 None 2026-04-21 01:08:26,294 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:08:26,317 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 01:08:26,319 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:08:26,336 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e42afa3551964617338d41 HTTP/1.1" 200 None 2026-04-21 01:08:26,337 - lava_callback - ERROR - Error processing callback for node 69e42afa3551964617338d41: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:43162 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38160 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46396 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51088 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36460 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55254 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:55268 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51256 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51272 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33708 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:09:46,271 - lava_callback - INFO - Received callback for node 69e42b453551964617338d61 INFO: 172.20.0.1:39582 - "POST /node/69e42b453551964617338d61 HTTP/1.0" 202 Accepted 2026-04-21 01:09:46,277 - lava_callback - INFO - Starting async job submit for node 69e42b453551964617338d61 2026-04-21 01:09:46,277 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:09:46,291 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42b453551964617338d61 HTTP/1.1" 200 None 2026-04-21 01:09:46,292 - lava_callback - DEBUG - Processing node 69e42b453551964617338d61: baseline-arm64-baylibre 2026-04-21 01:09:46,292 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e42b453551964617338d61 2026-04-21 01:09:46,293 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:09:46,311 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42b453551964617338d61 HTTP/1.1" 200 None 2026-04-21 01:09:46,312 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:09:46,326 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 01:09:46,327 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:09:46,345 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e42b453551964617338d61 HTTP/1.1" 200 None 2026-04-21 01:09:46,346 - lava_callback - ERROR - Error processing callback for node 69e42b453551964617338d61: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:39586 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54934 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40800 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36788 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:36794 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45140 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45152 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57094 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:10:46,459 - lava_callback - INFO - Received callback for node 69e42b913551964617338d87 INFO: 172.20.0.1:33594 - "POST /node/69e42b913551964617338d87 HTTP/1.0" 202 Accepted 2026-04-21 01:10:46,466 - lava_callback - INFO - Starting async job submit for node 69e42b913551964617338d87 2026-04-21 01:10:46,467 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:10:46,480 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42b913551964617338d87 HTTP/1.1" 200 None 2026-04-21 01:10:46,481 - lava_callback - DEBUG - Processing node 69e42b913551964617338d87: baseline-arm64-baylibre 2026-04-21 01:10:46,481 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e42b913551964617338d87 2026-04-21 01:10:46,482 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:10:46,502 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42b913551964617338d87 HTTP/1.1" 200 None 2026-04-21 01:10:46,503 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:10:46,518 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None INFO: 172.20.0.1:33596 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:10:46,521 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:10:46,544 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e42b913551964617338d87 HTTP/1.1" 200 None 2026-04-21 01:10:46,546 - lava_callback - ERROR - Error processing callback for node 69e42b913551964617338d87: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:46804 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33822 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39202 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39218 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51080 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51094 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49196 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39742 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50138 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:12:06,521 - lava_callback - INFO - Received callback for node 69e42bde3551964617338da5 INFO: 172.20.0.1:50972 - "POST /node/69e42bde3551964617338da5 HTTP/1.0" 202 Accepted 2026-04-21 01:12:06,526 - lava_callback - INFO - Starting async job submit for node 69e42bde3551964617338da5 2026-04-21 01:12:06,528 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:12:06,542 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42bde3551964617338da5 HTTP/1.1" 200 None 2026-04-21 01:12:06,543 - lava_callback - DEBUG - Processing node 69e42bde3551964617338da5: baseline-arm64-baylibre 2026-04-21 01:12:06,543 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e42bde3551964617338da5 2026-04-21 01:12:06,544 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 INFO: 172.20.0.1:50974 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:12:06,558 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42bde3551964617338da5 HTTP/1.1" 200 None 2026-04-21 01:12:06,559 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:12:06,572 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 01:12:06,573 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:12:06,590 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e42bde3551964617338da5 HTTP/1.1" 200 None 2026-04-21 01:12:06,591 - lava_callback - ERROR - Error processing callback for node 69e42bde3551964617338da5: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:38122 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:38134 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36818 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36820 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40242 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33916 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:12:53,940 - lava_callback - INFO - Received callback for node 69e6ceef60f5826562c15f61 INFO: 172.20.0.1:53528 - "POST /node/69e6ceef60f5826562c15f61 HTTP/1.0" 202 Accepted 2026-04-21 01:12:53,946 - lava_callback - INFO - Starting async job submit for node 69e6ceef60f5826562c15f61 2026-04-21 01:12:53,965 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:12:53,979 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6ceef60f5826562c15f61 HTTP/1.1" 200 None 2026-04-21 01:12:53,980 - lava_callback - DEBUG - Processing node 69e6ceef60f5826562c15f61: baseline-x86 2026-04-21 01:12:54,166 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:12:54,273 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:12:54,275 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-x86-69e6ceef60f5826562c15f61/log.txt.gz 2026-04-21 01:12:54,385 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:12:54,532 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:12:54,533 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69e6ceef60f5826562c15f61/lava-logs.yaml 2026-04-21 01:12:54,539 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:12:54,682 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:12:54,684 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69e6ceef60f5826562c15f61/lava_callback.json.gz 2026-04-21 01:12:54,720 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:12:54,733 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6ceef60f5826562c15f61 HTTP/1.1" 200 None 2026-04-21 01:12:54,734 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:12:54,747 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c23c60f5826562c150b4 HTTP/1.1" 200 None 2026-04-21 01:12:54,749 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:12:54,782 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6ceef60f5826562c15f61 HTTP/1.1" 200 None 2026-04-21 01:12:54,784 - lava_callback - INFO - Completed processing callback for node 69e6ceef60f5826562c15f61 2026-04-21 01:12:55,069 - lava_callback - INFO - Received callback for node 69e6cef260f5826562c15f6a INFO: 172.20.0.1:53534 - "POST /node/69e6cef260f5826562c15f6a HTTP/1.0" 202 Accepted 2026-04-21 01:12:55,080 - lava_callback - INFO - Starting async job submit for node 69e6cef260f5826562c15f6a 2026-04-21 01:12:55,098 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:12:55,120 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cef260f5826562c15f6a HTTP/1.1" 200 None 2026-04-21 01:12:55,121 - lava_callback - DEBUG - Processing node 69e6cef260f5826562c15f6a: baseline-x86 2026-04-21 01:12:55,330 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:12:55,430 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:12:55,432 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-x86-69e6cef260f5826562c15f6a/log.txt.gz 2026-04-21 01:12:55,540 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:12:55,656 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:12:55,658 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69e6cef260f5826562c15f6a/lava-logs.yaml 2026-04-21 01:12:55,664 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:12:55,746 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:12:55,748 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69e6cef260f5826562c15f6a/lava_callback.json.gz 2026-04-21 01:12:55,766 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:12:55,780 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cef260f5826562c15f6a HTTP/1.1" 200 None 2026-04-21 01:12:55,781 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:12:55,794 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c23c60f5826562c150b4 HTTP/1.1" 200 None 2026-04-21 01:12:55,795 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:12:55,815 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cef260f5826562c15f6a HTTP/1.1" 200 None 2026-04-21 01:12:55,816 - lava_callback - INFO - Completed processing callback for node 69e6cef260f5826562c15f6a INFO: 172.20.0.1:53550 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50748 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:13:14,887 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:13:14,905 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:50406 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50420 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44358 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44374 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:13:26,764 - lava_callback - INFO - Received callback for node 69e42c2a3551964617338dc9 INFO: 172.20.0.1:44382 - "POST /node/69e42c2a3551964617338dc9 HTTP/1.0" 202 Accepted 2026-04-21 01:13:26,770 - lava_callback - INFO - Starting async job submit for node 69e42c2a3551964617338dc9 2026-04-21 01:13:26,771 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:13:26,791 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42c2a3551964617338dc9 HTTP/1.1" 200 None 2026-04-21 01:13:26,792 - lava_callback - DEBUG - Processing node 69e42c2a3551964617338dc9: baseline-arm64-baylibre 2026-04-21 01:13:26,792 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e42c2a3551964617338dc9 2026-04-21 01:13:26,793 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:13:26,817 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42c2a3551964617338dc9 HTTP/1.1" 200 None 2026-04-21 01:13:26,819 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:13:26,838 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 01:13:26,840 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:13:26,877 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e42c2a3551964617338dc9 HTTP/1.1" 200 None 2026-04-21 01:13:26,878 - lava_callback - ERROR - Error processing callback for node 69e42c2a3551964617338dc9: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:34956 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51052 - "GET /node/694025af575bc060ce154141%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:51058 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45706 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47742 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41536 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41552 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60952 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60968 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42696 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42308 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:14:46,549 - lava_callback - INFO - Received callback for node 69e42c763551964617338de9 INFO: 172.20.0.1:42318 - "POST /node/69e42c763551964617338de9 HTTP/1.0" 202 Accepted 2026-04-21 01:14:46,559 - lava_callback - INFO - Starting async job submit for node 69e42c763551964617338de9 2026-04-21 01:14:46,560 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:14:46,581 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42c763551964617338de9 HTTP/1.1" 200 None 2026-04-21 01:14:46,583 - lava_callback - DEBUG - Processing node 69e42c763551964617338de9: baseline-arm64-baylibre 2026-04-21 01:14:46,583 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e42c763551964617338de9 2026-04-21 01:14:46,584 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:14:46,601 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42c763551964617338de9 HTTP/1.1" 200 None 2026-04-21 01:14:46,603 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:14:46,620 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 01:14:46,622 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:14:46,642 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e42c763551964617338de9 HTTP/1.1" 200 None 2026-04-21 01:14:46,643 - lava_callback - ERROR - Error processing callback for node 69e42c763551964617338de9: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:35564 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59374 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43978 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43988 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50390 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50396 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54982 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46212 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:15:46,812 - lava_callback - INFO - Received callback for node 69e42cbf3551964617338e0d INFO: 172.20.0.1:46220 - "POST /node/69e42cbf3551964617338e0d HTTP/1.0" 202 Accepted 2026-04-21 01:15:46,818 - lava_callback - INFO - Starting async job submit for node 69e42cbf3551964617338e0d 2026-04-21 01:15:46,819 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:15:46,833 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42cbf3551964617338e0d HTTP/1.1" 200 None 2026-04-21 01:15:46,834 - lava_callback - DEBUG - Processing node 69e42cbf3551964617338e0d: baseline-arm64-baylibre 2026-04-21 01:15:46,834 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e42cbf3551964617338e0d 2026-04-21 01:15:46,835 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:15:46,848 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42cbf3551964617338e0d HTTP/1.1" 200 None 2026-04-21 01:15:46,850 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:15:46,862 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 01:15:46,864 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:15:46,880 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e42cbf3551964617338e0d HTTP/1.1" 200 None 2026-04-21 01:15:46,881 - lava_callback - ERROR - Error processing callback for node 69e42cbf3551964617338e0d: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:35616 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60244 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57518 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57534 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38262 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38278 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42208 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34696 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59230 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56486 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:17:13,998 - lava_callback - INFO - Received callback for node 69e42d123551964617338e35 INFO: 172.20.0.1:40942 - "POST /node/69e42d123551964617338e35 HTTP/1.0" 202 Accepted 2026-04-21 01:17:14,004 - lava_callback - INFO - Starting async job submit for node 69e42d123551964617338e35 2026-04-21 01:17:14,005 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:17:14,021 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42d123551964617338e35 HTTP/1.1" 200 None 2026-04-21 01:17:14,022 - lava_callback - DEBUG - Processing node 69e42d123551964617338e35: baseline-arm64-clabbe 2026-04-21 01:17:14,022 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e42d123551964617338e35 2026-04-21 01:17:14,023 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:17:14,035 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42d123551964617338e35 HTTP/1.1" 200 None 2026-04-21 01:17:14,036 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:17:14,054 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 01:17:14,056 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:17:14,073 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e42d123551964617338e35 HTTP/1.1" 200 None 2026-04-21 01:17:14,074 - lava_callback - ERROR - Error processing callback for node 69e42d123551964617338e35: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:40944 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40948 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40958 - "GET /node/6991865210b45da7d7d0fd40%5Cn HTTP/1.0" 405 Method Not Allowed 2026-04-21 01:17:20,125 - lava_callback - INFO - Received callback for node 69e6d00860f5826562c15ff8 INFO: 172.20.0.1:40772 - "POST /node/69e6d00860f5826562c15ff8 HTTP/1.0" 202 Accepted 2026-04-21 01:17:20,131 - lava_callback - INFO - Starting async job submit for node 69e6d00860f5826562c15ff8 2026-04-21 01:17:20,133 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:17:20,159 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6d00860f5826562c15ff8 HTTP/1.1" 200 None 2026-04-21 01:17:20,160 - lava_callback - DEBUG - Processing node 69e6d00860f5826562c15ff8: baseline-nfs-arm64 2026-04-21 01:17:20,161 - lava_callback - WARNING - Failed to upload log 2026-04-21 01:17:20,162 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:17:20,281 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:17:20,282 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6d00860f5826562c15ff8/lava-logs.yaml 2026-04-21 01:17:20,284 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:17:20,401 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:17:20,403 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6d00860f5826562c15ff8/lava_callback.json.gz 2026-04-21 01:17:20,404 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:17:20,418 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6d00860f5826562c15ff8 HTTP/1.1" 200 None 2026-04-21 01:17:20,419 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:17:20,433 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 01:17:20,435 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:17:20,460 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6d00860f5826562c15ff8 HTTP/1.1" 200 None 2026-04-21 01:17:20,466 - lava_callback - INFO - Completed processing callback for node 69e6d00860f5826562c15ff8 INFO: 172.20.0.1:40776 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40782 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:17:34,228 - lava_callback - INFO - Received callback for node 69e6cb8260f5826562c15c07 INFO: 172.20.0.1:53912 - "POST /node/69e6cb8260f5826562c15c07 HTTP/1.0" 202 Accepted 2026-04-21 01:17:34,237 - lava_callback - INFO - Starting async job submit for node 69e6cb8260f5826562c15c07 2026-04-21 01:17:34,281 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:17:34,295 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb8260f5826562c15c07 HTTP/1.1" 200 None 2026-04-21 01:17:34,296 - lava_callback - DEBUG - Processing node 69e6cb8260f5826562c15c07: baseline-nfs-arm64 2026-04-21 01:17:35,455 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:17:35,775 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:17:35,777 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb8260f5826562c15c07/log.txt.gz 2026-04-21 01:17:36,381 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.20.0.1:53920 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:17:36,591 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:17:36,594 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb8260f5826562c15c07/lava-logs.yaml 2026-04-21 01:17:36,616 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:17:36,770 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:17:36,773 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb8260f5826562c15c07/lava_callback.json.gz DEBUG: Setup failed for 69e6cb8260f5826562c15c07. Transitting job node result: incomplete -> fail 2026-04-21 01:17:36,862 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:17:36,875 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb8260f5826562c15c07 HTTP/1.1" 200 None 2026-04-21 01:17:36,877 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:17:36,889 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 01:17:36,891 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:17:36,912 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb8260f5826562c15c07 HTTP/1.1" 200 None 2026-04-21 01:17:36,996 - lava_callback - INFO - Completed processing callback for node 69e6cb8260f5826562c15c07 2026-04-21 01:17:44,908 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:17:44,923 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:51094 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:17:52,082 - lava_callback - INFO - Received callback for node 69e6cb7960f5826562c15bee INFO: 172.20.0.1:46890 - "POST /node/69e6cb7960f5826562c15bee HTTP/1.0" 202 Accepted 2026-04-21 01:17:52,088 - lava_callback - INFO - Starting async job submit for node 69e6cb7960f5826562c15bee 2026-04-21 01:17:52,113 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:17:52,129 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7960f5826562c15bee HTTP/1.1" 200 None 2026-04-21 01:17:52,130 - lava_callback - DEBUG - Processing node 69e6cb7960f5826562c15bee: baseline-arm64 2026-04-21 01:17:52,732 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:17:52,737 - lava_callback - INFO - Received callback for node 69e6cb8460f5826562c15c0d INFO: 172.20.0.1:46906 - "POST /node/69e6cb8460f5826562c15c0d HTTP/1.0" 202 Accepted 2026-04-21 01:17:52,746 - lava_callback - INFO - Starting async job submit for node 69e6cb8460f5826562c15c0d 2026-04-21 01:17:52,791 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:17:52,804 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb8460f5826562c15c0d HTTP/1.1" 200 None 2026-04-21 01:17:52,805 - lava_callback - DEBUG - Processing node 69e6cb8460f5826562c15c0d: baseline-nfs-arm64 2026-04-21 01:17:53,104 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:17:53,152 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7960f5826562c15bee/log.txt.gz 2026-04-21 01:17:53,768 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:17:54,030 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:17:54,074 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7960f5826562c15bee/lava-logs.yaml 2026-04-21 01:17:54,177 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:17:54,273 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:17:54,369 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:17:54,371 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7960f5826562c15bee/lava_callback.json.gz 2026-04-21 01:17:54,395 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:17:54,411 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7960f5826562c15bee HTTP/1.1" 200 None 2026-04-21 01:17:54,413 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:17:54,426 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 01:17:54,428 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:17:54,445 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb7960f5826562c15bee HTTP/1.1" 200 None 2026-04-21 01:17:54,498 - lava_callback - INFO - Completed processing callback for node 69e6cb7960f5826562c15bee 2026-04-21 01:17:54,627 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:17:54,629 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb8460f5826562c15c0d/log.txt.gz 2026-04-21 01:17:55,259 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:17:55,434 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:17:55,436 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb8460f5826562c15c0d/lava-logs.yaml 2026-04-21 01:17:55,461 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:17:55,605 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:17:55,607 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb8460f5826562c15c0d/lava_callback.json.gz DEBUG: Setup failed for 69e6cb8460f5826562c15c0d. Transitting job node result: incomplete -> fail 2026-04-21 01:17:55,693 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:17:55,707 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb8460f5826562c15c0d HTTP/1.1" 200 None 2026-04-21 01:17:55,708 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:17:55,723 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 01:17:55,725 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:17:55,744 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb8460f5826562c15c0d HTTP/1.1" 200 None 2026-04-21 01:17:55,826 - lava_callback - INFO - Completed processing callback for node 69e6cb8460f5826562c15c0d INFO: 172.20.0.1:46910 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50160 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:18:14,925 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:18:14,946 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:56190 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:56198 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46608 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46622 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:18:26,803 - lava_callback - INFO - Received callback for node 69e42d583551964617338e4e INFO: 172.20.0.1:46628 - "POST /node/69e42d583551964617338e4e HTTP/1.0" 202 Accepted 2026-04-21 01:18:26,809 - lava_callback - INFO - Starting async job submit for node 69e42d583551964617338e4e 2026-04-21 01:18:26,810 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:18:26,823 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42d583551964617338e4e HTTP/1.1" 200 None 2026-04-21 01:18:26,824 - lava_callback - DEBUG - Processing node 69e42d583551964617338e4e: baseline-arm64-baylibre 2026-04-21 01:18:26,824 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e42d583551964617338e4e 2026-04-21 01:18:26,825 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:18:26,859 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42d583551964617338e4e HTTP/1.1" 200 None 2026-04-21 01:18:26,860 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:18:26,873 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 01:18:26,875 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:18:26,898 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e42d583551964617338e4e HTTP/1.1" 200 None 2026-04-21 01:18:26,899 - lava_callback - ERROR - Error processing callback for node 69e42d583551964617338e4e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-04-21 01:18:30,296 - lava_callback - INFO - Received callback for node 69e6d05160f5826562c16025 INFO: 172.20.0.1:40296 - "POST /node/69e6d05160f5826562c16025 HTTP/1.0" 202 Accepted 2026-04-21 01:18:30,302 - lava_callback - INFO - Starting async job submit for node 69e6d05160f5826562c16025 2026-04-21 01:18:30,306 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:18:30,324 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6d05160f5826562c16025 HTTP/1.1" 200 None 2026-04-21 01:18:30,325 - lava_callback - DEBUG - Processing node 69e6d05160f5826562c16025: baseline-arm 2026-04-21 01:18:30,332 - lava_callback - WARNING - Failed to upload log 2026-04-21 01:18:30,335 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:18:30,474 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:18:30,476 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69e6d05160f5826562c16025/lava-logs.yaml 2026-04-21 01:18:30,477 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:18:30,597 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:18:30,599 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69e6d05160f5826562c16025/lava_callback.json.gz 2026-04-21 01:18:30,602 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:18:30,616 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6d05160f5826562c16025 HTTP/1.1" 200 None 2026-04-21 01:18:30,617 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:18:30,630 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 01:18:30,631 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:18:30,654 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6d05160f5826562c16025 HTTP/1.1" 200 None 2026-04-21 01:18:30,665 - lava_callback - INFO - Completed processing callback for node 69e6d05160f5826562c16025 INFO: 172.20.0.1:40312 - "GET /node/695404aa9c6929bd9fb6f35a%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:40316 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:18:44,948 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:18:44,965 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:38294 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:18:46,889 - lava_callback - INFO - Received callback for node 69e6cb7160f5826562c15bd5 INFO: 172.20.0.1:38310 - "POST /node/69e6cb7160f5826562c15bd5 HTTP/1.0" 202 Accepted 2026-04-21 01:18:46,895 - lava_callback - INFO - Starting async job submit for node 69e6cb7160f5826562c15bd5 2026-04-21 01:18:46,915 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:18:46,931 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7160f5826562c15bd5 HTTP/1.1" 200 None 2026-04-21 01:18:46,932 - lava_callback - DEBUG - Processing node 69e6cb7160f5826562c15bd5: baseline-arm64 2026-04-21 01:18:47,139 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:18:47,255 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:18:47,257 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7160f5826562c15bd5/log.txt.gz 2026-04-21 01:18:47,389 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:18:47,533 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:18:47,535 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7160f5826562c15bd5/lava-logs.yaml 2026-04-21 01:18:47,542 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:18:47,646 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:18:47,648 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6cb7160f5826562c15bd5/lava_callback.json.gz DEBUG: Setup failed for 69e6cb7160f5826562c15bd5. Transitting job node result: incomplete -> fail 2026-04-21 01:18:47,708 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:18:47,730 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7160f5826562c15bd5 HTTP/1.1" 200 None 2026-04-21 01:18:47,731 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:18:47,745 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 01:18:47,747 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:18:47,769 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb7160f5826562c15bd5 HTTP/1.1" 200 None 2026-04-21 01:18:47,815 - lava_callback - INFO - Completed processing callback for node 69e6cb7160f5826562c15bd5 INFO: 172.20.0.1:40892 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58374 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:19:14,910 - lava_callback - INFO - Received callback for node 69e5ea94ab69ed7ddbbc3b59 INFO: 172.20.0.1:39078 - "POST /node/69e5ea94ab69ed7ddbbc3b59 HTTP/1.0" 202 Accepted 2026-04-21 01:19:14,916 - lava_callback - INFO - Starting async job submit for node 69e5ea94ab69ed7ddbbc3b59 2026-04-21 01:19:14,933 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:19:14,949 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e5ea94ab69ed7ddbbc3b59 HTTP/1.1" 200 None 2026-04-21 01:19:14,950 - lava_callback - DEBUG - Processing node 69e5ea94ab69ed7ddbbc3b59: baseline-x86 2026-04-21 01:19:14,977 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:19:15,074 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:39090 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 01:19:15,306 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:19:15,570 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:19:15,572 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-x86-69e5ea94ab69ed7ddbbc3b59/log.txt.gz 2026-04-21 01:19:15,739 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:19:15,918 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:19:15,920 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69e5ea94ab69ed7ddbbc3b59/lava-logs.yaml 2026-04-21 01:19:15,928 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:19:16,024 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:19:16,025 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69e5ea94ab69ed7ddbbc3b59/lava_callback.json.gz 2026-04-21 01:19:16,043 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:19:16,067 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e5ea94ab69ed7ddbbc3b59 HTTP/1.1" 200 None 2026-04-21 01:19:16,068 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:19:16,082 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e5e150ab69ed7ddbbc37ee HTTP/1.1" 200 None 2026-04-21 01:19:16,083 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:19:16,106 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e5ea94ab69ed7ddbbc3b59 HTTP/1.1" 200 None 2026-04-21 01:19:16,107 - lava_callback - INFO - Completed processing callback for node 69e5ea94ab69ed7ddbbc3b59 INFO: 172.20.0.1:39094 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34376 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34386 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34500 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:19:45,082 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:19:45,098 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:50270 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:19:51,221 - lava_callback - INFO - Received callback for node 69e6d07560f5826562c1603a INFO: 172.20.0.1:58412 - "POST /node/69e6d07560f5826562c1603a HTTP/1.0" 202 Accepted 2026-04-21 01:19:51,228 - lava_callback - INFO - Starting async job submit for node 69e6d07560f5826562c1603a 2026-04-21 01:19:51,258 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:19:51,272 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6d07560f5826562c1603a HTTP/1.1" 200 None 2026-04-21 01:19:51,273 - lava_callback - DEBUG - Processing node 69e6d07560f5826562c1603a: baseline-arm64 2026-04-21 01:19:51,405 - lava_callback - WARNING - Failed to upload log 2026-04-21 01:19:51,481 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:19:51,812 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:19:51,814 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6d07560f5826562c1603a/lava-logs.yaml 2026-04-21 01:19:51,817 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:19:51,920 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:19:51,922 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6d07560f5826562c1603a/lava_callback.json.gz 2026-04-21 01:19:51,953 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:19:51,967 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6d07560f5826562c1603a HTTP/1.1" 200 None 2026-04-21 01:19:51,968 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:19:51,981 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 01:19:51,983 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:19:52,003 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6d07560f5826562c1603a HTTP/1.1" 200 None 2026-04-21 01:19:52,082 - lava_callback - INFO - Completed processing callback for node 69e6d07560f5826562c1603a 2026-04-21 01:19:54,114 - lava_callback - INFO - Received callback for node 69e42da93551964617338e76 INFO: 172.20.0.1:58418 - "POST /node/69e42da93551964617338e76 HTTP/1.0" 202 Accepted 2026-04-21 01:19:54,120 - lava_callback - INFO - Starting async job submit for node 69e42da93551964617338e76 2026-04-21 01:19:54,121 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:19:54,137 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42da93551964617338e76 HTTP/1.1" 200 None 2026-04-21 01:19:54,138 - lava_callback - DEBUG - Processing node 69e42da93551964617338e76: baseline-arm64-clabbe 2026-04-21 01:19:54,138 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e42da93551964617338e76 2026-04-21 01:19:54,138 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:19:54,154 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42da93551964617338e76 HTTP/1.1" 200 None 2026-04-21 01:19:54,156 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:19:54,169 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 01:19:54,171 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:19:54,191 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e42da93551964617338e76 HTTP/1.1" 200 None 2026-04-21 01:19:54,192 - lava_callback - ERROR - Error processing callback for node 69e42da93551964617338e76: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:58432 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41140 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:20:15,100 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:20:15,115 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:43948 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43964 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40964 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40970 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35930 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40068 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58388 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57330 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:21:06,960 - lava_callback - INFO - Received callback for node 69e42def3551964617338e94 INFO: 172.20.0.1:57336 - "POST /node/69e42def3551964617338e94 HTTP/1.0" 202 Accepted 2026-04-21 01:21:06,965 - lava_callback - INFO - Starting async job submit for node 69e42def3551964617338e94 2026-04-21 01:21:06,966 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:21:06,980 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42def3551964617338e94 HTTP/1.1" 200 None 2026-04-21 01:21:06,981 - lava_callback - DEBUG - Processing node 69e42def3551964617338e94: baseline-arm64-baylibre 2026-04-21 01:21:06,982 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e42def3551964617338e94 2026-04-21 01:21:06,982 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:21:06,995 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42def3551964617338e94 HTTP/1.1" 200 None 2026-04-21 01:21:06,997 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:21:07,011 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 01:21:07,012 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:21:07,030 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e42def3551964617338e94 HTTP/1.1" 200 None 2026-04-21 01:21:07,031 - lava_callback - ERROR - Error processing callback for node 69e42def3551964617338e94: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-04-21 01:21:11,600 - lava_callback - INFO - Received callback for node 69e6cb7f60f5826562c15c01 INFO: 172.20.0.1:47016 - "POST /node/69e6cb7f60f5826562c15c01 HTTP/1.0" 202 Accepted 2026-04-21 01:21:11,607 - lava_callback - INFO - Starting async job submit for node 69e6cb7f60f5826562c15c01 2026-04-21 01:21:11,634 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:21:11,649 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7f60f5826562c15c01 HTTP/1.1" 200 None 2026-04-21 01:21:11,650 - lava_callback - DEBUG - Processing node 69e6cb7f60f5826562c15c01: baseline-nfs-arm64 2026-04-21 01:21:11,937 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:21:12,216 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:21:12,218 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb7f60f5826562c15c01/log.txt.gz 2026-04-21 01:21:12,414 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:21:12,566 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:21:12,569 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb7f60f5826562c15c01/lava-logs.yaml 2026-04-21 01:21:12,576 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:21:12,693 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:21:12,694 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb7f60f5826562c15c01/lava_callback.json.gz DEBUG: Setup failed for 69e6cb7f60f5826562c15c01. Transitting job node result: incomplete -> fail 2026-04-21 01:21:12,739 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:21:12,754 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb7f60f5826562c15c01 HTTP/1.1" 200 None 2026-04-21 01:21:12,755 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:21:12,775 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 01:21:12,776 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:21:12,807 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb7f60f5826562c15c01 HTTP/1.1" 200 None 2026-04-21 01:21:12,857 - lava_callback - INFO - Completed processing callback for node 69e6cb7f60f5826562c15c01 2026-04-21 01:21:15,117 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:21:15,136 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:47032 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47034 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33578 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:21:22,839 - lava_callback - INFO - Received callback for node 69e6d0c660f5826562c16065 INFO: 172.20.0.1:33580 - "POST /node/69e6d0c660f5826562c16065 HTTP/1.0" 202 Accepted 2026-04-21 01:21:22,846 - lava_callback - INFO - Starting async job submit for node 69e6d0c660f5826562c16065 2026-04-21 01:21:22,883 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:21:22,898 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6d0c660f5826562c16065 HTTP/1.1" 200 None 2026-04-21 01:21:22,899 - lava_callback - DEBUG - Processing node 69e6d0c660f5826562c16065: baseline-arm64 2026-04-21 01:21:23,108 - lava_callback - WARNING - Failed to upload log 2026-04-21 01:21:23,226 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:21:23,430 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:21:23,432 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6d0c660f5826562c16065/lava-logs.yaml 2026-04-21 01:21:23,436 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:21:23,542 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:21:23,543 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6d0c660f5826562c16065/lava_callback.json.gz 2026-04-21 01:21:23,589 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:21:23,604 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6d0c660f5826562c16065 HTTP/1.1" 200 None 2026-04-21 01:21:23,606 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:21:23,621 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 01:21:23,623 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:21:23,650 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6d0c660f5826562c16065 HTTP/1.1" 200 None 2026-04-21 01:21:23,729 - lava_callback - INFO - Completed processing callback for node 69e6d0c660f5826562c16065 INFO: 172.20.0.1:33592 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60126 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:21:45,139 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:21:45,154 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:56852 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36876 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45998 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:22:14,203 - lava_callback - INFO - Received callback for node 69e42e403551964617338ebc INFO: 172.20.0.1:37450 - "POST /node/69e42e403551964617338ebc HTTP/1.0" 202 Accepted 2026-04-21 01:22:14,210 - lava_callback - INFO - Starting async job submit for node 69e42e403551964617338ebc 2026-04-21 01:22:14,212 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:22:14,228 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42e403551964617338ebc HTTP/1.1" 200 None 2026-04-21 01:22:14,229 - lava_callback - DEBUG - Processing node 69e42e403551964617338ebc: baseline-arm64-clabbe 2026-04-21 01:22:14,229 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e42e403551964617338ebc 2026-04-21 01:22:14,230 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:22:14,248 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42e403551964617338ebc HTTP/1.1" 200 None 2026-04-21 01:22:14,250 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:22:14,264 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 01:22:14,266 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:22:14,283 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e42e403551964617338ebc HTTP/1.1" 200 None 2026-04-21 01:22:14,284 - lava_callback - ERROR - Error processing callback for node 69e42e403551964617338ebc: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:37456 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37470 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59642 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59656 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45010 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40164 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43090 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56500 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39106 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39114 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:23:20,409 - lava_callback - INFO - Received callback for node 69e6cb8760f5826562c15c3f INFO: 172.20.0.1:57506 - "POST /node/69e6cb8760f5826562c15c3f HTTP/1.0" 202 Accepted 2026-04-21 01:23:20,416 - lava_callback - INFO - Starting async job submit for node 69e6cb8760f5826562c15c3f 2026-04-21 01:23:20,442 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:23:20,464 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb8760f5826562c15c3f HTTP/1.1" 200 None 2026-04-21 01:23:20,465 - lava_callback - DEBUG - Processing node 69e6cb8760f5826562c15c3f: baseline-nfs-arm64 INFO: 172.20.0.1:57508 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:23:20,903 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:23:20,985 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:23:20,987 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb8760f5826562c15c3f/log.txt.gz 2026-04-21 01:23:21,180 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:23:21,333 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:23:21,335 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb8760f5826562c15c3f/lava-logs.yaml 2026-04-21 01:23:21,345 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:23:21,473 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:23:21,475 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6cb8760f5826562c15c3f/lava_callback.json.gz 2026-04-21 01:23:21,501 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:23:21,515 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb8760f5826562c15c3f HTTP/1.1" 200 None 2026-04-21 01:23:21,516 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:23:21,530 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 01:23:21,531 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:23:21,554 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb8760f5826562c15c3f HTTP/1.1" 200 None 2026-04-21 01:23:21,617 - lava_callback - INFO - Completed processing callback for node 69e6cb8760f5826562c15c3f INFO: 172.20.0.1:57512 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:23:27,021 - lava_callback - INFO - Received callback for node 69e42e863551964617338ed6 INFO: 172.20.0.1:57516 - "POST /node/69e42e863551964617338ed6 HTTP/1.0" 202 Accepted 2026-04-21 01:23:27,026 - lava_callback - INFO - Starting async job submit for node 69e42e863551964617338ed6 2026-04-21 01:23:27,028 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:23:27,041 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42e863551964617338ed6 HTTP/1.1" 200 None 2026-04-21 01:23:27,042 - lava_callback - DEBUG - Processing node 69e42e863551964617338ed6: baseline-arm64-baylibre 2026-04-21 01:23:27,042 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e42e863551964617338ed6 2026-04-21 01:23:27,043 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:23:27,061 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42e863551964617338ed6 HTTP/1.1" 200 None 2026-04-21 01:23:27,063 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:23:27,076 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 01:23:27,078 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:23:27,097 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e42e863551964617338ed6 HTTP/1.1" 200 None 2026-04-21 01:23:27,098 - lava_callback - ERROR - Error processing callback for node 69e42e863551964617338ed6: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-04-21 01:23:27,146 - lava_callback - INFO - Received callback for node 69e6cd9a60f5826562c15ee9 INFO: 172.20.0.1:57518 - "POST /node/69e6cd9a60f5826562c15ee9 HTTP/1.0" 202 Accepted 2026-04-21 01:23:27,159 - lava_callback - INFO - Starting async job submit for node 69e6cd9a60f5826562c15ee9 2026-04-21 01:23:27,230 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:23:27,243 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cd9a60f5826562c15ee9 HTTP/1.1" 200 None 2026-04-21 01:23:27,244 - lava_callback - DEBUG - Processing node 69e6cd9a60f5826562c15ee9: baseline-arm 2026-04-21 01:23:27,854 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:23:27,935 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:23:27,937 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69e6cd9a60f5826562c15ee9/log.txt.gz 2026-04-21 01:23:28,288 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:23:28,482 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:23:28,485 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69e6cd9a60f5826562c15ee9/lava-logs.yaml 2026-04-21 01:23:28,499 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:23:28,618 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:23:28,620 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69e6cd9a60f5826562c15ee9/lava_callback.json.gz DEBUG: Setup failed for 69e6cd9a60f5826562c15ee9. Transitting job node result: incomplete -> fail 2026-04-21 01:23:28,736 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:23:28,778 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cd9a60f5826562c15ee9 HTTP/1.1" 200 None 2026-04-21 01:23:28,779 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:23:28,792 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 01:23:28,794 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:23:28,815 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cd9a60f5826562c15ee9 HTTP/1.1" 200 None 2026-04-21 01:23:28,918 - lava_callback - INFO - Completed processing callback for node 69e6cd9a60f5826562c15ee9 INFO: 172.20.0.1:39458 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:23:45,157 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:23:45,182 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:37052 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35174 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48942 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40026 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40032 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57576 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57592 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50970 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45878 - "GET /node/6989265c9fc35f969e1f1d27%5Cn HTTP/1.0" 405 Method Not Allowed 2026-04-21 01:24:43,339 - lava_callback - INFO - Received callback for node 69e6cdbe60f5826562c15ef3 INFO: 172.20.0.1:45882 - "POST /node/69e6cdbe60f5826562c15ef3 HTTP/1.0" 202 Accepted 2026-04-21 01:24:43,345 - lava_callback - INFO - Starting async job submit for node 69e6cdbe60f5826562c15ef3 2026-04-21 01:24:43,408 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:24:43,423 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cdbe60f5826562c15ef3 HTTP/1.1" 200 None 2026-04-21 01:24:43,424 - lava_callback - DEBUG - Processing node 69e6cdbe60f5826562c15ef3: ltp-smoke 2026-04-21 01:24:43,919 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:24:44,025 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:24:44,027 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-smoke-69e6cdbe60f5826562c15ef3/log.txt.gz 2026-04-21 01:24:44,289 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:24:44,604 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:24:44,606 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-smoke-69e6cdbe60f5826562c15ef3/lava-logs.yaml 2026-04-21 01:24:44,617 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:24:44,917 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:24:44,918 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-smoke-69e6cdbe60f5826562c15ef3/lava_callback.json.gz 2026-04-21 01:24:45,039 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:24:45,053 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cdbe60f5826562c15ef3 HTTP/1.1" 200 None 2026-04-21 01:24:45,055 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:24:45,068 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c21560f5826562c15091 HTTP/1.1" 200 None 2026-04-21 01:24:45,071 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:24:45,093 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cdbe60f5826562c15ef3 HTTP/1.1" 200 None 2026-04-21 01:24:45,210 - lava_callback - INFO - Completed processing callback for node 69e6cdbe60f5826562c15ef3 INFO: 172.20.0.1:45890 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:24:54,325 - lava_callback - INFO - Received callback for node 69e42ed53551964617338efd INFO: 172.20.0.1:33306 - "POST /node/69e42ed53551964617338efd HTTP/1.0" 202 Accepted 2026-04-21 01:24:54,330 - lava_callback - INFO - Starting async job submit for node 69e42ed53551964617338efd 2026-04-21 01:24:54,331 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:24:54,345 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42ed53551964617338efd HTTP/1.1" 200 None 2026-04-21 01:24:54,346 - lava_callback - DEBUG - Processing node 69e42ed53551964617338efd: baseline-arm64-clabbe 2026-04-21 01:24:54,346 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e42ed53551964617338efd 2026-04-21 01:24:54,347 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:24:54,362 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42ed53551964617338efd HTTP/1.1" 200 None 2026-04-21 01:24:54,363 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:24:54,383 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 01:24:54,384 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:24:54,402 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e42ed53551964617338efd HTTP/1.1" 200 None 2026-04-21 01:24:54,403 - lava_callback - ERROR - Error processing callback for node 69e42ed53551964617338efd: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:33322 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42948 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51584 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 01:25:15,190 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:25:15,209 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:51600 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60570 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60584 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39420 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34626 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44638 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37348 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:26:14,390 - lava_callback - INFO - Received callback for node 69e42f203551964617338f16 INFO: 172.20.0.1:37718 - "POST /node/69e42f203551964617338f16 HTTP/1.0" 202 Accepted 2026-04-21 01:26:14,396 - lava_callback - INFO - Starting async job submit for node 69e42f203551964617338f16 2026-04-21 01:26:14,397 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:26:14,411 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42f203551964617338f16 HTTP/1.1" 200 None 2026-04-21 01:26:14,412 - lava_callback - DEBUG - Processing node 69e42f203551964617338f16: baseline-arm64-clabbe 2026-04-21 01:26:14,413 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e42f203551964617338f16 2026-04-21 01:26:14,413 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:26:14,426 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42f203551964617338f16 HTTP/1.1" 200 None 2026-04-21 01:26:14,428 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:26:14,441 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 01:26:14,443 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:26:14,463 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e42f203551964617338f16 HTTP/1.1" 200 None 2026-04-21 01:26:14,465 - lava_callback - ERROR - Error processing callback for node 69e42f203551964617338f16: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:37734 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37742 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55514 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55524 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43046 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42210 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:26:46,670 - lava_callback - INFO - Received callback for node 69e64109ab69ed7ddbbc52b5 INFO: 172.20.0.1:42220 - "POST /node/69e64109ab69ed7ddbbc52b5 HTTP/1.0" 202 Accepted 2026-04-21 01:26:46,679 - lava_callback - INFO - Starting async job submit for node 69e64109ab69ed7ddbbc52b5 2026-04-21 01:26:46,709 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:26:46,728 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e64109ab69ed7ddbbc52b5 HTTP/1.1" 200 None 2026-04-21 01:26:46,729 - lava_callback - DEBUG - Processing node 69e64109ab69ed7ddbbc52b5: baseline-x86 2026-04-21 01:26:47,094 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:26:47,359 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:26:47,361 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-x86-69e64109ab69ed7ddbbc52b5/log.txt.gz 2026-04-21 01:26:47,576 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:26:47,761 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:26:47,763 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69e64109ab69ed7ddbbc52b5/lava-logs.yaml 2026-04-21 01:26:47,771 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:26:47,890 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:26:47,893 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69e64109ab69ed7ddbbc52b5/lava_callback.json.gz 2026-04-21 01:26:47,917 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:26:47,939 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e64109ab69ed7ddbbc52b5 HTTP/1.1" 200 None 2026-04-21 01:26:47,941 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:26:47,957 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e5e150ab69ed7ddbbc37ee HTTP/1.1" 200 None 2026-04-21 01:26:47,959 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:26:47,984 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e64109ab69ed7ddbbc52b5 HTTP/1.1" 200 None 2026-04-21 01:26:47,986 - lava_callback - INFO - Completed processing callback for node 69e64109ab69ed7ddbbc52b5 INFO: 172.20.0.1:33002 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36002 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:27:07,280 - lava_callback - INFO - Received callback for node 69e42f613551964617338f33 INFO: 172.20.0.1:36006 - "POST /node/69e42f613551964617338f33 HTTP/1.0" 202 Accepted 2026-04-21 01:27:07,285 - lava_callback - INFO - Starting async job submit for node 69e42f613551964617338f33 2026-04-21 01:27:07,286 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:27:07,301 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42f613551964617338f33 HTTP/1.1" 200 None 2026-04-21 01:27:07,302 - lava_callback - DEBUG - Processing node 69e42f613551964617338f33: baseline-arm64-baylibre 2026-04-21 01:27:07,302 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e42f613551964617338f33 2026-04-21 01:27:07,303 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:27:07,336 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42f613551964617338f33 HTTP/1.1" 200 None 2026-04-21 01:27:07,338 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:27:07,353 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc360e5dbfa74e32f96891 HTTP/1.1" 200 None 2026-04-21 01:27:07,354 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:27:07,386 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e42f613551964617338f33 HTTP/1.1" 200 None 2026-04-21 01:27:07,388 - lava_callback - ERROR - Error processing callback for node 69e42f613551964617338f33: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:58660 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 01:27:15,211 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:27:15,236 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:58664 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35052 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35068 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32806 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40430 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55288 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35386 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59866 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59880 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34968 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34984 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:28:34,479 - lava_callback - INFO - Received callback for node 69e42fb53551964617338f55 INFO: 172.20.0.1:37342 - "POST /node/69e42fb53551964617338f55 HTTP/1.0" 202 Accepted 2026-04-21 01:28:34,485 - lava_callback - INFO - Starting async job submit for node 69e42fb53551964617338f55 2026-04-21 01:28:34,486 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:28:34,500 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42fb53551964617338f55 HTTP/1.1" 200 None 2026-04-21 01:28:34,501 - lava_callback - DEBUG - Processing node 69e42fb53551964617338f55: baseline-arm64-clabbe 2026-04-21 01:28:34,501 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e42fb53551964617338f55 2026-04-21 01:28:34,502 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:28:34,517 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e42fb53551964617338f55 HTTP/1.1" 200 None 2026-04-21 01:28:34,519 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:28:34,533 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 01:28:34,535 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:28:34,554 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e42fb53551964617338f55 HTTP/1.1" 200 None 2026-04-21 01:28:34,556 - lava_callback - ERROR - Error processing callback for node 69e42fb53551964617338f55: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:37358 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51160 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48460 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57722 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42556 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:42566 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56262 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56272 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58412 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59132 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:29:54,525 - lava_callback - INFO - Received callback for node 69e430023551964617338f79 INFO: 172.20.0.1:48056 - "POST /node/69e430023551964617338f79 HTTP/1.0" 202 Accepted 2026-04-21 01:29:54,531 - lava_callback - INFO - Starting async job submit for node 69e430023551964617338f79 2026-04-21 01:29:54,532 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:29:54,546 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e430023551964617338f79 HTTP/1.1" 200 None 2026-04-21 01:29:54,547 - lava_callback - DEBUG - Processing node 69e430023551964617338f79: baseline-arm64-clabbe 2026-04-21 01:29:54,547 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e430023551964617338f79 2026-04-21 01:29:54,548 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:29:54,561 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e430023551964617338f79 HTTP/1.1" 200 None 2026-04-21 01:29:54,562 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:29:54,582 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 01:29:54,584 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:29:54,601 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e430023551964617338f79 HTTP/1.1" 200 None 2026-04-21 01:29:54,602 - lava_callback - ERROR - Error processing callback for node 69e430023551964617338f79: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:48062 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55114 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34822 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34838 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48824 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48832 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46490 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34678 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33632 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52480 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:31:14,576 - lava_callback - INFO - Received callback for node 69e4304e3551964617338f92 INFO: 172.20.0.1:35364 - "POST /node/69e4304e3551964617338f92 HTTP/1.0" 202 Accepted 2026-04-21 01:31:14,582 - lava_callback - INFO - Starting async job submit for node 69e4304e3551964617338f92 2026-04-21 01:31:14,583 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:31:14,597 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4304e3551964617338f92 HTTP/1.1" 200 None 2026-04-21 01:31:14,598 - lava_callback - DEBUG - Processing node 69e4304e3551964617338f92: baseline-arm64-clabbe 2026-04-21 01:31:14,598 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4304e3551964617338f92 2026-04-21 01:31:14,599 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:31:14,611 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4304e3551964617338f92 HTTP/1.1" 200 None 2026-04-21 01:31:14,613 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:31:14,634 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 01:31:14,635 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:31:14,654 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4304e3551964617338f92 HTTP/1.1" 200 None 2026-04-21 01:31:14,656 - lava_callback - ERROR - Error processing callback for node 69e4304e3551964617338f92: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:35374 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35380 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56920 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56936 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49582 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:31:38,372 - lava_callback - INFO - Received callback for node 69e65eca47863bb428fe4cd0 INFO: 172.20.0.1:49598 - "POST /node/69e65eca47863bb428fe4cd0 HTTP/1.0" 202 Accepted 2026-04-21 01:31:38,378 - lava_callback - INFO - Starting async job submit for node 69e65eca47863bb428fe4cd0 2026-04-21 01:31:38,395 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:31:38,414 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e65eca47863bb428fe4cd0 HTTP/1.1" 200 None 2026-04-21 01:31:38,415 - lava_callback - DEBUG - Processing node 69e65eca47863bb428fe4cd0: baseline-x86 2026-04-21 01:31:38,724 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:31:39,019 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:31:39,021 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-x86-69e65eca47863bb428fe4cd0/log.txt.gz 2026-04-21 01:31:39,190 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:31:39,370 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:31:39,372 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69e65eca47863bb428fe4cd0/lava-logs.yaml 2026-04-21 01:31:39,380 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:31:39,504 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:31:39,506 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69e65eca47863bb428fe4cd0/lava_callback.json.gz 2026-04-21 01:31:39,523 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:31:39,543 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e65eca47863bb428fe4cd0 HTTP/1.1" 200 None 2026-04-21 01:31:39,544 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:31:39,557 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6523c47863bb428fe4b0e HTTP/1.1" 200 None 2026-04-21 01:31:39,558 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:31:39,578 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e65eca47863bb428fe4cd0 HTTP/1.1" 200 None 2026-04-21 01:31:39,579 - lava_callback - INFO - Completed processing callback for node 69e65eca47863bb428fe4cd0 2026-04-21 01:31:45,240 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:31:45,258 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:59312 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47274 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46504 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:32:14,721 - lava_callback - INFO - Received callback for node 69e430993551964617338fb7 INFO: 172.20.0.1:54340 - "POST /node/69e430993551964617338fb7 HTTP/1.0" 202 Accepted 2026-04-21 01:32:14,727 - lava_callback - INFO - Starting async job submit for node 69e430993551964617338fb7 2026-04-21 01:32:14,728 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:32:14,753 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e430993551964617338fb7 HTTP/1.1" 200 None 2026-04-21 01:32:14,754 - lava_callback - DEBUG - Processing node 69e430993551964617338fb7: baseline-arm64-clabbe 2026-04-21 01:32:14,754 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e430993551964617338fb7 2026-04-21 01:32:14,755 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:32:14,776 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e430993551964617338fb7 HTTP/1.1" 200 None 2026-04-21 01:32:14,778 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:32:14,792 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 01:32:14,794 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:32:14,821 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e430993551964617338fb7 HTTP/1.1" 200 None 2026-04-21 01:32:14,822 - lava_callback - ERROR - Error processing callback for node 69e430993551964617338fb7: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54356 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:54372 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38644 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38652 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35982 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55808 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56188 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34000 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44690 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44696 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49450 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49456 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:33:27,646 - lava_callback - INFO - Received callback for node 69e430dd3551964617338fd0 INFO: 172.20.0.1:49466 - "POST /node/69e430dd3551964617338fd0 HTTP/1.0" 202 Accepted 2026-04-21 01:33:27,651 - lava_callback - INFO - Starting async job submit for node 69e430dd3551964617338fd0 2026-04-21 01:33:27,652 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:33:27,668 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e430dd3551964617338fd0 HTTP/1.1" 200 None 2026-04-21 01:33:27,669 - lava_callback - DEBUG - Processing node 69e430dd3551964617338fd0: baseline-arm64-baylibre 2026-04-21 01:33:27,670 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e430dd3551964617338fd0 2026-04-21 01:33:27,670 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:33:27,683 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e430dd3551964617338fd0 HTTP/1.1" 200 None 2026-04-21 01:33:27,685 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:33:27,706 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc360e5dbfa74e32f96891 HTTP/1.1" 200 None 2026-04-21 01:33:27,707 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:33:27,725 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e430dd3551964617338fd0 HTTP/1.1" 200 None 2026-04-21 01:33:27,727 - lava_callback - ERROR - Error processing callback for node 69e430dd3551964617338fd0: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49588 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39790 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38408 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55730 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50494 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50504 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36322 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36326 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51388 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51924 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:34:54,762 - lava_callback - INFO - Received callback for node 69e4312e3551964617338ff4 INFO: 172.20.0.1:45506 - "POST /node/69e4312e3551964617338ff4 HTTP/1.0" 202 Accepted 2026-04-21 01:34:54,768 - lava_callback - INFO - Starting async job submit for node 69e4312e3551964617338ff4 2026-04-21 01:34:54,769 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:34:54,783 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4312e3551964617338ff4 HTTP/1.1" 200 None 2026-04-21 01:34:54,784 - lava_callback - DEBUG - Processing node 69e4312e3551964617338ff4: baseline-arm64-clabbe 2026-04-21 01:34:54,784 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4312e3551964617338ff4 2026-04-21 01:34:54,785 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:34:54,797 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4312e3551964617338ff4 HTTP/1.1" 200 None 2026-04-21 01:34:54,799 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:34:54,811 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 01:34:54,813 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:34:54,830 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4312e3551964617338ff4 HTTP/1.1" 200 None 2026-04-21 01:34:54,831 - lava_callback - ERROR - Error processing callback for node 69e4312e3551964617338ff4: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:45514 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59936 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46396 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46406 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:35:18,230 - lava_callback - INFO - Received callback for node 69e6a27edc041e577b77f12e INFO: 172.20.0.1:46412 - "POST /node/69e6a27edc041e577b77f12e HTTP/1.0" 202 Accepted 2026-04-21 01:35:18,235 - lava_callback - INFO - Starting async job submit for node 69e6a27edc041e577b77f12e 2026-04-21 01:35:18,253 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:35:18,267 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6a27edc041e577b77f12e HTTP/1.1" 200 None 2026-04-21 01:35:18,268 - lava_callback - DEBUG - Processing node 69e6a27edc041e577b77f12e: baseline-x86 2026-04-21 01:35:18,588 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:35:18,861 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:35:18,863 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-x86-69e6a27edc041e577b77f12e/log.txt.gz 2026-04-21 01:35:19,026 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:35:19,195 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:35:19,197 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69e6a27edc041e577b77f12e/lava-logs.yaml 2026-04-21 01:35:19,205 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:35:19,328 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:35:19,329 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69e6a27edc041e577b77f12e/lava_callback.json.gz 2026-04-21 01:35:19,348 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:35:19,364 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6a27edc041e577b77f12e HTTP/1.1" 200 None 2026-04-21 01:35:19,366 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:35:19,386 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e5e150ab69ed7ddbbc37ee HTTP/1.1" 200 None 2026-04-21 01:35:19,388 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:35:19,409 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6a27edc041e577b77f12e HTTP/1.1" 200 None 2026-04-21 01:35:19,410 - lava_callback - INFO - Completed processing callback for node 69e6a27edc041e577b77f12e INFO: 172.20.0.1:44436 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44452 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54734 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:35:45,261 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:35:45,279 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:43790 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46196 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37768 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:36:07,711 - lava_callback - INFO - Received callback for node 69e43174355196461733900d INFO: 172.20.0.1:37772 - "POST /node/69e43174355196461733900d HTTP/1.0" 202 Accepted 2026-04-21 01:36:07,716 - lava_callback - INFO - Starting async job submit for node 69e43174355196461733900d 2026-04-21 01:36:07,717 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:36:07,731 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43174355196461733900d HTTP/1.1" 200 None 2026-04-21 01:36:07,731 - lava_callback - DEBUG - Processing node 69e43174355196461733900d: baseline-arm64-baylibre 2026-04-21 01:36:07,732 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e43174355196461733900d 2026-04-21 01:36:07,732 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:36:07,745 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43174355196461733900d HTTP/1.1" 200 None 2026-04-21 01:36:07,746 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:36:07,766 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc360e5dbfa74e32f96891 HTTP/1.1" 200 None 2026-04-21 01:36:07,767 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:36:07,787 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e43174355196461733900d HTTP/1.1" 200 None 2026-04-21 01:36:07,789 - lava_callback - ERROR - Error processing callback for node 69e43174355196461733900d: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:34752 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34758 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42050 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42052 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46752 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60030 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32858 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45844 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:37:14,817 - lava_callback - INFO - Received callback for node 69e431c43551964617339032 INFO: 172.20.0.1:37880 - "POST /node/69e431c43551964617339032 HTTP/1.0" 202 Accepted 2026-04-21 01:37:14,823 - lava_callback - INFO - Starting async job submit for node 69e431c43551964617339032 2026-04-21 01:37:14,824 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:37:14,838 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e431c43551964617339032 HTTP/1.1" 200 None 2026-04-21 01:37:14,839 - lava_callback - DEBUG - Processing node 69e431c43551964617339032: baseline-arm64-clabbe 2026-04-21 01:37:14,839 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e431c43551964617339032 2026-04-21 01:37:14,840 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:37:14,865 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e431c43551964617339032 HTTP/1.1" 200 None 2026-04-21 01:37:14,867 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:37:14,881 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 01:37:14,883 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:37:14,914 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e431c43551964617339032 HTTP/1.1" 200 None 2026-04-21 01:37:14,915 - lava_callback - ERROR - Error processing callback for node 69e431c43551964617339032: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:37882 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37894 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47920 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47930 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35796 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54342 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52842 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59466 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35590 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35606 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48658 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48674 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:38:27,812 - lava_callback - INFO - Received callback for node 69e43206355196461733904b INFO: 172.20.0.1:48688 - "POST /node/69e43206355196461733904b HTTP/1.0" 202 Accepted 2026-04-21 01:38:27,817 - lava_callback - INFO - Starting async job submit for node 69e43206355196461733904b 2026-04-21 01:38:27,818 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:38:27,832 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43206355196461733904b HTTP/1.1" 200 None 2026-04-21 01:38:27,833 - lava_callback - DEBUG - Processing node 69e43206355196461733904b: baseline-arm64-baylibre 2026-04-21 01:38:27,834 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e43206355196461733904b 2026-04-21 01:38:27,834 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:38:27,847 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43206355196461733904b HTTP/1.1" 200 None 2026-04-21 01:38:27,848 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:38:27,862 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc360e5dbfa74e32f96891 HTTP/1.1" 200 None 2026-04-21 01:38:27,863 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:38:27,883 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e43206355196461733904b HTTP/1.1" 200 None 2026-04-21 01:38:27,884 - lava_callback - ERROR - Error processing callback for node 69e43206355196461733904b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:57200 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58988 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54626 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48290 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51154 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51170 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48090 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48094 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42818 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38198 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:39:54,971 - lava_callback - INFO - Received callback for node 69e432593551964617339070 INFO: 172.20.0.1:41158 - "POST /node/69e432593551964617339070 HTTP/1.0" 202 Accepted 2026-04-21 01:39:54,977 - lava_callback - INFO - Starting async job submit for node 69e432593551964617339070 2026-04-21 01:39:54,978 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:39:54,992 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e432593551964617339070 HTTP/1.1" 200 None 2026-04-21 01:39:54,993 - lava_callback - DEBUG - Processing node 69e432593551964617339070: baseline-arm64-clabbe 2026-04-21 01:39:54,993 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e432593551964617339070 2026-04-21 01:39:54,994 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:39:55,007 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e432593551964617339070 HTTP/1.1" 200 None 2026-04-21 01:39:55,008 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:39:55,021 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 01:39:55,022 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:39:55,039 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e432593551964617339070 HTTP/1.1" 200 None 2026-04-21 01:39:55,040 - lava_callback - ERROR - Error processing callback for node 69e432593551964617339070: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:41162 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57296 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50632 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50638 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34464 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34470 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34142 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52184 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:40:47,982 - lava_callback - INFO - Received callback for node 69e4329c355196461733908c INFO: 172.20.0.1:52200 - "POST /node/69e4329c355196461733908c HTTP/1.0" 202 Accepted 2026-04-21 01:40:47,992 - lava_callback - INFO - Starting async job submit for node 69e4329c355196461733908c 2026-04-21 01:40:47,993 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:40:48,009 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4329c355196461733908c HTTP/1.1" 200 None 2026-04-21 01:40:48,010 - lava_callback - DEBUG - Processing node 69e4329c355196461733908c: baseline-arm64-baylibre 2026-04-21 01:40:48,010 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4329c355196461733908c 2026-04-21 01:40:48,011 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:40:48,035 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4329c355196461733908c HTTP/1.1" 200 None 2026-04-21 01:40:48,036 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:40:48,049 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc360e5dbfa74e32f96891 HTTP/1.1" 200 None 2026-04-21 01:40:48,050 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:40:48,067 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4329c355196461733908c HTTP/1.1" 200 None 2026-04-21 01:40:48,069 - lava_callback - ERROR - Error processing callback for node 69e4329c355196461733908c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:38492 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45886 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41076 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41080 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45184 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45186 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36280 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53570 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43768 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53204 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:42:08,031 - lava_callback - INFO - Received callback for node 69e432e835519646173390a8 INFO: 172.20.0.1:53210 - "POST /node/69e432e835519646173390a8 HTTP/1.0" 202 Accepted 2026-04-21 01:42:08,036 - lava_callback - INFO - Starting async job submit for node 69e432e835519646173390a8 2026-04-21 01:42:08,037 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:42:08,052 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e432e835519646173390a8 HTTP/1.1" 200 None 2026-04-21 01:42:08,053 - lava_callback - DEBUG - Processing node 69e432e835519646173390a8: baseline-arm64-baylibre 2026-04-21 01:42:08,053 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e432e835519646173390a8 2026-04-21 01:42:08,054 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:42:08,066 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e432e835519646173390a8 HTTP/1.1" 200 None 2026-04-21 01:42:08,067 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:42:08,080 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc360e5dbfa74e32f96891 HTTP/1.1" 200 None 2026-04-21 01:42:08,082 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:42:08,101 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e432e835519646173390a8 HTTP/1.1" 200 None 2026-04-21 01:42:08,102 - lava_callback - ERROR - Error processing callback for node 69e432e835519646173390a8: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:46964 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46978 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48454 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48460 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52876 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42010 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51260 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46794 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35506 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35508 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49758 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49768 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:43:35,099 - lava_callback - INFO - Received callback for node 69e4333835519646173390ca INFO: 172.20.0.1:40922 - "POST /node/69e4333835519646173390ca HTTP/1.0" 202 Accepted 2026-04-21 01:43:35,105 - lava_callback - INFO - Starting async job submit for node 69e4333835519646173390ca 2026-04-21 01:43:35,105 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:43:35,121 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4333835519646173390ca HTTP/1.1" 200 None 2026-04-21 01:43:35,122 - lava_callback - DEBUG - Processing node 69e4333835519646173390ca: baseline-arm64-clabbe 2026-04-21 01:43:35,122 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4333835519646173390ca 2026-04-21 01:43:35,123 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:43:35,136 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4333835519646173390ca HTTP/1.1" 200 None 2026-04-21 01:43:35,137 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:43:35,150 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 01:43:35,151 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:43:35,167 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4333835519646173390ca HTTP/1.1" 200 None 2026-04-21 01:43:35,169 - lava_callback - ERROR - Error processing callback for node 69e4333835519646173390ca: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:40928 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41444 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47512 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51422 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59568 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59570 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58390 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58396 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52096 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46796 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:44:55,207 - lava_callback - INFO - Received callback for node 69e4338435519646173390f1 INFO: 172.20.0.1:49362 - "POST /node/69e4338435519646173390f1 HTTP/1.0" 202 Accepted 2026-04-21 01:44:55,213 - lava_callback - INFO - Starting async job submit for node 69e4338435519646173390f1 2026-04-21 01:44:55,214 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:44:55,228 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4338435519646173390f1 HTTP/1.1" 200 None 2026-04-21 01:44:55,229 - lava_callback - DEBUG - Processing node 69e4338435519646173390f1: baseline-arm64-clabbe 2026-04-21 01:44:55,230 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4338435519646173390f1 2026-04-21 01:44:55,230 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:44:55,243 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4338435519646173390f1 HTTP/1.1" 200 None 2026-04-21 01:44:55,245 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:44:55,258 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 01:44:55,259 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:44:55,277 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4338435519646173390f1 HTTP/1.1" 200 None 2026-04-21 01:44:55,278 - lava_callback - ERROR - Error processing callback for node 69e4338435519646173390f1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49372 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58518 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34380 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34392 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45240 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45250 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39572 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43484 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:45:48,169 - lava_callback - INFO - Received callback for node 69e433c7355196461733910a INFO: 172.20.0.1:43498 - "POST /node/69e433c7355196461733910a HTTP/1.0" 202 Accepted 2026-04-21 01:45:48,174 - lava_callback - INFO - Starting async job submit for node 69e433c7355196461733910a 2026-04-21 01:45:48,175 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:45:48,190 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e433c7355196461733910a HTTP/1.1" 200 None 2026-04-21 01:45:48,190 - lava_callback - DEBUG - Processing node 69e433c7355196461733910a: baseline-arm64-baylibre 2026-04-21 01:45:48,191 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e433c7355196461733910a 2026-04-21 01:45:48,191 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:45:48,204 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e433c7355196461733910a HTTP/1.1" 200 None 2026-04-21 01:45:48,206 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:45:48,220 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded948f490930782fb6e2d HTTP/1.1" 200 None 2026-04-21 01:45:48,221 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:45:48,238 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e433c7355196461733910a HTTP/1.1" 200 None 2026-04-21 01:45:48,239 - lava_callback - ERROR - Error processing callback for node 69e433c7355196461733910a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:35474 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59644 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48304 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48316 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38110 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38118 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42220 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59294 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57606 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59550 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:47:08,310 - lava_callback - INFO - Received callback for node 69e434113551964617339126 INFO: 172.20.0.1:59566 - "POST /node/69e434113551964617339126 HTTP/1.0" 202 Accepted 2026-04-21 01:47:08,315 - lava_callback - INFO - Starting async job submit for node 69e434113551964617339126 2026-04-21 01:47:08,316 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:47:08,330 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e434113551964617339126 HTTP/1.1" 200 None 2026-04-21 01:47:08,331 - lava_callback - DEBUG - Processing node 69e434113551964617339126: baseline-arm64-baylibre 2026-04-21 01:47:08,331 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e434113551964617339126 2026-04-21 01:47:08,332 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:47:08,346 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e434113551964617339126 HTTP/1.1" 200 None 2026-04-21 01:47:08,347 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:47:08,361 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded948f490930782fb6e2d HTTP/1.1" 200 None 2026-04-21 01:47:08,363 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:47:08,380 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e434113551964617339126 HTTP/1.1" 200 None 2026-04-21 01:47:08,382 - lava_callback - ERROR - Error processing callback for node 69e434113551964617339126: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54908 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:54924 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55816 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55828 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36180 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47636 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54358 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46330 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60520 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60526 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54406 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54414 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:48:28,320 - lava_callback - INFO - Received callback for node 69e4345b355196461733914a INFO: 172.20.0.1:54430 - "POST /node/69e4345b355196461733914a HTTP/1.0" 202 Accepted 2026-04-21 01:48:28,326 - lava_callback - INFO - Starting async job submit for node 69e4345b355196461733914a 2026-04-21 01:48:28,327 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:48:28,340 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4345b355196461733914a HTTP/1.1" 200 None 2026-04-21 01:48:28,341 - lava_callback - DEBUG - Processing node 69e4345b355196461733914a: baseline-arm64-baylibre 2026-04-21 01:48:28,342 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4345b355196461733914a 2026-04-21 01:48:28,342 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:48:28,360 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4345b355196461733914a HTTP/1.1" 200 None 2026-04-21 01:48:28,362 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:48:28,375 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded948f490930782fb6e2d HTTP/1.1" 200 None 2026-04-21 01:48:28,376 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:48:28,393 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4345b355196461733914a HTTP/1.1" 200 None 2026-04-21 01:48:28,394 - lava_callback - ERROR - Error processing callback for node 69e4345b355196461733914a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:55590 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38008 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44626 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34324 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:49:09,532 - lava_callback - INFO - Received callback for node 69e6d66760f5826562c16314 INFO: 172.20.0.1:34326 - "POST /node/69e6d66760f5826562c16314 HTTP/1.0" 202 Accepted 2026-04-21 01:49:09,540 - lava_callback - INFO - Starting async job submit for node 69e6d66760f5826562c16314 2026-04-21 01:49:09,575 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:49:09,594 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6d66760f5826562c16314 HTTP/1.1" 200 None 2026-04-21 01:49:09,595 - lava_callback - DEBUG - Processing node 69e6d66760f5826562c16314: baseline-arm64 2026-04-21 01:49:10,027 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:49:10,136 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:49:10,138 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6d66760f5826562c16314/log.txt.gz 2026-04-21 01:49:10,334 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:49:10,691 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:49:10,693 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6d66760f5826562c16314/lava-logs.yaml 2026-04-21 01:49:10,701 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:49:10,830 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:49:10,832 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6d66760f5826562c16314/lava_callback.json.gz DEBUG: Setup failed for 69e6d66760f5826562c16314. Transitting job node result: incomplete -> fail 2026-04-21 01:49:10,902 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:49:10,916 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6d66760f5826562c16314 HTTP/1.1" 200 None 2026-04-21 01:49:10,918 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:49:10,931 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 01:49:10,932 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:49:10,951 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6d66760f5826562c16314 HTTP/1.1" 200 None 2026-04-21 01:49:11,020 - lava_callback - INFO - Completed processing callback for node 69e6d66760f5826562c16314 INFO: 172.20.0.1:49740 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 01:49:15,283 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:49:15,298 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:49746 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38714 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38718 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54950 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41640 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:49:48,438 - lava_callback - INFO - Received callback for node 69e434a73551964617339167 INFO: 172.20.0.1:41646 - "POST /node/69e434a73551964617339167 HTTP/1.0" 202 Accepted 2026-04-21 01:49:48,444 - lava_callback - INFO - Starting async job submit for node 69e434a73551964617339167 2026-04-21 01:49:48,445 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:49:48,459 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e434a73551964617339167 HTTP/1.1" 200 None 2026-04-21 01:49:48,460 - lava_callback - DEBUG - Processing node 69e434a73551964617339167: baseline-arm64-baylibre 2026-04-21 01:49:48,460 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e434a73551964617339167 2026-04-21 01:49:48,461 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:49:48,474 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e434a73551964617339167 HTTP/1.1" 200 None 2026-04-21 01:49:48,475 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:49:48,488 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded948f490930782fb6e2d HTTP/1.1" 200 None 2026-04-21 01:49:48,489 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:49:48,507 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e434a73551964617339167 HTTP/1.1" 200 None 2026-04-21 01:49:48,508 - lava_callback - ERROR - Error processing callback for node 69e434a73551964617339167: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54102 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59522 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51308 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51312 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35502 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35508 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40088 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36490 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:50:48,360 - lava_callback - INFO - Received callback for node 69e434f23551964617339188 INFO: 172.20.0.1:36496 - "POST /node/69e434f23551964617339188 HTTP/1.0" 202 Accepted 2026-04-21 01:50:48,365 - lava_callback - INFO - Starting async job submit for node 69e434f23551964617339188 2026-04-21 01:50:48,366 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:50:48,380 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e434f23551964617339188 HTTP/1.1" 200 None 2026-04-21 01:50:48,381 - lava_callback - DEBUG - Processing node 69e434f23551964617339188: baseline-arm64-baylibre 2026-04-21 01:50:48,381 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e434f23551964617339188 2026-04-21 01:50:48,382 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:50:48,411 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e434f23551964617339188 HTTP/1.1" 200 None 2026-04-21 01:50:48,413 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:50:48,430 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded948f490930782fb6e2d HTTP/1.1" 200 None 2026-04-21 01:50:48,432 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:50:48,453 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e434f23551964617339188 HTTP/1.1" 200 None 2026-04-21 01:50:48,454 - lava_callback - ERROR - Error processing callback for node 69e434f23551964617339188: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54222 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49648 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50750 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50760 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39932 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39948 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49830 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59038 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47980 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47986 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52628 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:52:08,474 - lava_callback - INFO - Received callback for node 69e4353d35519646173391a5 INFO: 172.20.0.1:52642 - "POST /node/69e4353d35519646173391a5 HTTP/1.0" 202 Accepted 2026-04-21 01:52:08,480 - lava_callback - INFO - Starting async job submit for node 69e4353d35519646173391a5 2026-04-21 01:52:08,481 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:52:08,504 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4353d35519646173391a5 HTTP/1.1" 200 None 2026-04-21 01:52:08,505 - lava_callback - DEBUG - Processing node 69e4353d35519646173391a5: baseline-arm64-baylibre 2026-04-21 01:52:08,505 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4353d35519646173391a5 2026-04-21 01:52:08,506 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:52:08,520 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4353d35519646173391a5 HTTP/1.1" 200 None 2026-04-21 01:52:08,521 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:52:08,536 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded948f490930782fb6e2d HTTP/1.1" 200 None 2026-04-21 01:52:08,538 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:52:08,555 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4353d35519646173391a5 HTTP/1.1" 200 None 2026-04-21 01:52:08,556 - lava_callback - ERROR - Error processing callback for node 69e4353d35519646173391a5: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:42580 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:42594 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54684 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54690 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54702 - "GET /favicon.ico HTTP/1.0" 404 Not Found INFO: 172.20.0.1:48160 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58478 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59170 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53722 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38666 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:38668 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38098 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38106 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:53:35,474 - lava_callback - INFO - Received callback for node 69e4358c35519646173391c7 INFO: 172.20.0.1:36316 - "POST /node/69e4358c35519646173391c7 HTTP/1.0" 202 Accepted 2026-04-21 01:53:35,480 - lava_callback - INFO - Starting async job submit for node 69e4358c35519646173391c7 2026-04-21 01:53:35,481 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:53:35,494 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4358c35519646173391c7 HTTP/1.1" 200 None 2026-04-21 01:53:35,495 - lava_callback - DEBUG - Processing node 69e4358c35519646173391c7: baseline-arm64-clabbe 2026-04-21 01:53:35,495 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4358c35519646173391c7 2026-04-21 01:53:35,496 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:53:35,508 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4358c35519646173391c7 HTTP/1.1" 200 None 2026-04-21 01:53:35,510 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:53:35,523 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc360e5dbfa74e32f96891 HTTP/1.1" 200 None 2026-04-21 01:53:35,525 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:53:35,549 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4358c35519646173391c7 HTTP/1.1" 200 None 2026-04-21 01:53:35,551 - lava_callback - ERROR - Error processing callback for node 69e4358c35519646173391c7: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:36318 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52158 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45446 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53070 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40964 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40978 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44542 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44544 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:54:35,474 - lava_callback - INFO - Received callback for node 69e435d835519646173391e3 INFO: 172.20.0.1:57012 - "POST /node/69e435d835519646173391e3 HTTP/1.0" 202 Accepted 2026-04-21 01:54:35,479 - lava_callback - INFO - Starting async job submit for node 69e435d835519646173391e3 2026-04-21 01:54:35,480 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:54:35,494 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e435d835519646173391e3 HTTP/1.1" 200 None 2026-04-21 01:54:35,495 - lava_callback - DEBUG - Processing node 69e435d835519646173391e3: baseline-arm64-clabbe 2026-04-21 01:54:35,495 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e435d835519646173391e3 2026-04-21 01:54:35,496 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:54:35,512 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e435d835519646173391e3 HTTP/1.1" 200 None 2026-04-21 01:54:35,514 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:54:35,526 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded948f490930782fb6e2d HTTP/1.1" 200 None 2026-04-21 01:54:35,528 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:54:35,546 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e435d835519646173391e3 HTTP/1.1" 200 None 2026-04-21 01:54:35,547 - lava_callback - ERROR - Error processing callback for node 69e435d835519646173391e3: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:57018 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52326 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37730 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51238 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37910 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37914 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48808 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48824 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50250 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42576 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:55:55,540 - lava_callback - INFO - Received callback for node 69e436233551964617339205 INFO: 172.20.0.1:57030 - "POST /node/69e436233551964617339205 HTTP/1.0" 202 Accepted 2026-04-21 01:55:55,547 - lava_callback - INFO - Starting async job submit for node 69e436233551964617339205 2026-04-21 01:55:55,548 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:55:55,570 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e436233551964617339205 HTTP/1.1" 200 None 2026-04-21 01:55:55,571 - lava_callback - DEBUG - Processing node 69e436233551964617339205: baseline-arm64-clabbe 2026-04-21 01:55:55,571 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e436233551964617339205 2026-04-21 01:55:55,572 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:55:55,585 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e436233551964617339205 HTTP/1.1" 200 None 2026-04-21 01:55:55,586 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:55:55,599 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded948f490930782fb6e2d HTTP/1.1" 200 None 2026-04-21 01:55:55,601 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:55:55,618 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e436233551964617339205 HTTP/1.1" 200 None 2026-04-21 01:55:55,619 - lava_callback - ERROR - Error processing callback for node 69e436233551964617339205: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:57044 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37256 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37964 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37972 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49438 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49452 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33532 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47656 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56804 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60806 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46372 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 01:57:15,632 - lava_callback - INFO - Received callback for node 69e4366d3551964617339221 INFO: 172.20.0.1:46382 - "POST /node/69e4366d3551964617339221 HTTP/1.0" 202 Accepted 2026-04-21 01:57:15,638 - lava_callback - INFO - Starting async job submit for node 69e4366d3551964617339221 2026-04-21 01:57:15,639 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:57:15,653 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4366d3551964617339221 HTTP/1.1" 200 None 2026-04-21 01:57:15,654 - lava_callback - DEBUG - Processing node 69e4366d3551964617339221: baseline-arm64-clabbe 2026-04-21 01:57:15,654 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4366d3551964617339221 2026-04-21 01:57:15,655 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:57:15,668 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4366d3551964617339221 HTTP/1.1" 200 None 2026-04-21 01:57:15,669 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:57:15,682 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded948f490930782fb6e2d HTTP/1.1" 200 None 2026-04-21 01:57:15,683 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:57:15,702 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4366d3551964617339221 HTTP/1.1" 200 None 2026-04-21 01:57:15,703 - lava_callback - ERROR - Error processing callback for node 69e4366d3551964617339221: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:46394 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41264 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41268 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43640 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48684 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38228 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50570 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48766 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48772 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53438 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53442 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:58:35,664 - lava_callback - INFO - Received callback for node 69e436b83551964617339242 INFO: 172.20.0.1:43084 - "POST /node/69e436b83551964617339242 HTTP/1.0" 202 Accepted 2026-04-21 01:58:35,670 - lava_callback - INFO - Starting async job submit for node 69e436b83551964617339242 2026-04-21 01:58:35,671 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:58:35,692 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e436b83551964617339242 HTTP/1.1" 200 None 2026-04-21 01:58:35,693 - lava_callback - DEBUG - Processing node 69e436b83551964617339242: baseline-arm64-clabbe 2026-04-21 01:58:35,693 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e436b83551964617339242 2026-04-21 01:58:35,693 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:58:35,710 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e436b83551964617339242 HTTP/1.1" 200 None 2026-04-21 01:58:35,712 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:58:35,747 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded948f490930782fb6e2d HTTP/1.1" 200 None 2026-04-21 01:58:35,748 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:58:35,769 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e436b83551964617339242 HTTP/1.1" 200 None 2026-04-21 01:58:35,770 - lava_callback - ERROR - Error processing callback for node 69e436b83551964617339242: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:43086 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:58:41,430 - lava_callback - INFO - Received callback for node 69e6d9b960f5826562c1645f INFO: 172.20.0.1:54552 - "POST /node/69e6d9b960f5826562c1645f HTTP/1.0" 202 Accepted 2026-04-21 01:58:41,436 - lava_callback - INFO - Starting async job submit for node 69e6d9b960f5826562c1645f 2026-04-21 01:58:41,438 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:58:41,452 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6d9b960f5826562c1645f HTTP/1.1" 200 None 2026-04-21 01:58:41,453 - lava_callback - DEBUG - Processing node 69e6d9b960f5826562c1645f: baseline-nfs-arm64 2026-04-21 01:58:41,454 - lava_callback - WARNING - Failed to upload log 2026-04-21 01:58:41,455 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:58:41,729 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:58:41,730 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6d9b960f5826562c1645f/lava-logs.yaml 2026-04-21 01:58:41,731 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 01:58:41,853 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 01:58:41,855 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6d9b960f5826562c1645f/lava_callback.json.gz 2026-04-21 01:58:41,857 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:58:41,871 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6d9b960f5826562c1645f HTTP/1.1" 200 None 2026-04-21 01:58:41,872 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:58:41,886 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 01:58:41,887 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:58:41,911 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6d9b960f5826562c1645f HTTP/1.1" 200 None 2026-04-21 01:58:41,920 - lava_callback - INFO - Completed processing callback for node 69e6d9b960f5826562c1645f 2026-04-21 01:58:45,302 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:58:45,321 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:54568 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34102 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50768 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59124 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59140 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36098 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36106 - "GET / HTTP/1.0" 200 OK 2026-04-21 01:59:35,690 - lava_callback - INFO - Received callback for node 69e43704355196461733925f INFO: 172.20.0.1:36098 - "POST /node/69e43704355196461733925f HTTP/1.0" 202 Accepted 2026-04-21 01:59:35,699 - lava_callback - INFO - Starting async job submit for node 69e43704355196461733925f 2026-04-21 01:59:35,700 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:59:35,718 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43704355196461733925f HTTP/1.1" 200 None 2026-04-21 01:59:35,719 - lava_callback - DEBUG - Processing node 69e43704355196461733925f: baseline-arm64-clabbe 2026-04-21 01:59:35,719 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e43704355196461733925f 2026-04-21 01:59:35,720 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:59:35,733 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43704355196461733925f HTTP/1.1" 200 None 2026-04-21 01:59:35,735 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:59:35,748 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded948f490930782fb6e2d HTTP/1.1" 200 None 2026-04-21 01:59:35,749 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 01:59:35,766 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e43704355196461733925f HTTP/1.1" 200 None 2026-04-21 01:59:35,767 - lava_callback - ERROR - Error processing callback for node 69e43704355196461733925f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:36112 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45434 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39334 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59668 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51936 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51948 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54434 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54442 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46972 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44280 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:00:55,761 - lava_callback - INFO - Received callback for node 69e437503551964617339281 INFO: 172.20.0.1:42978 - "POST /node/69e437503551964617339281 HTTP/1.0" 202 Accepted 2026-04-21 02:00:55,767 - lava_callback - INFO - Starting async job submit for node 69e437503551964617339281 2026-04-21 02:00:55,768 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:00:55,782 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e437503551964617339281 HTTP/1.1" 200 None 2026-04-21 02:00:55,783 - lava_callback - DEBUG - Processing node 69e437503551964617339281: baseline-arm64-clabbe 2026-04-21 02:00:55,783 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e437503551964617339281 2026-04-21 02:00:55,784 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:00:55,796 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e437503551964617339281 HTTP/1.1" 200 None 2026-04-21 02:00:55,798 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:00:55,811 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded948f490930782fb6e2d HTTP/1.1" 200 None 2026-04-21 02:00:55,812 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:00:55,837 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e437503551964617339281 HTTP/1.1" 200 None 2026-04-21 02:00:55,838 - lava_callback - ERROR - Error processing callback for node 69e437503551964617339281: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:42986 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50562 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47566 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47572 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38064 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38080 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48550 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44934 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36078 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55502 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39230 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 02:02:15,856 - lava_callback - INFO - Received callback for node 69e4379d355196461733929d INFO: 172.20.0.1:39242 - "POST /node/69e4379d355196461733929d HTTP/1.0" 202 Accepted 2026-04-21 02:02:15,862 - lava_callback - INFO - Starting async job submit for node 69e4379d355196461733929d 2026-04-21 02:02:15,863 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:02:15,877 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4379d355196461733929d HTTP/1.1" 200 None 2026-04-21 02:02:15,878 - lava_callback - DEBUG - Processing node 69e4379d355196461733929d: baseline-arm64-clabbe 2026-04-21 02:02:15,878 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4379d355196461733929d 2026-04-21 02:02:15,879 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:02:15,891 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4379d355196461733929d HTTP/1.1" 200 None 2026-04-21 02:02:15,893 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:02:15,906 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc360e5dbfa74e32f96891 HTTP/1.1" 200 None 2026-04-21 02:02:15,907 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:02:15,940 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4379d355196461733929d HTTP/1.1" 200 None 2026-04-21 02:02:15,941 - lava_callback - ERROR - Error processing callback for node 69e4379d355196461733929d: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:39246 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51636 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51640 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59558 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36050 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44596 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41362 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36538 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:36552 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37572 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37586 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:03:35,901 - lava_callback - INFO - Received callback for node 69e437e935519646173392bd INFO: 172.20.0.1:53970 - "POST /node/69e437e935519646173392bd HTTP/1.0" 202 Accepted 2026-04-21 02:03:35,909 - lava_callback - INFO - Starting async job submit for node 69e437e935519646173392bd 2026-04-21 02:03:35,911 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:03:35,926 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e437e935519646173392bd HTTP/1.1" 200 None 2026-04-21 02:03:35,928 - lava_callback - DEBUG - Processing node 69e437e935519646173392bd: baseline-arm64-clabbe 2026-04-21 02:03:35,928 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e437e935519646173392bd 2026-04-21 02:03:35,929 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:03:35,972 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e437e935519646173392bd HTTP/1.1" 200 None 2026-04-21 02:03:35,973 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:03:36,016 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc360e5dbfa74e32f96891 HTTP/1.1" 200 None 2026-04-21 02:03:36,018 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:03:36,045 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e437e935519646173392bd HTTP/1.1" 200 None 2026-04-21 02:03:36,046 - lava_callback - ERROR - Error processing callback for node 69e437e935519646173392bd: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:53982 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45030 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42422 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38198 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39956 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39964 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60904 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60912 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38952 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47678 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:04:55,978 - lava_callback - INFO - Received callback for node 69e4383435519646173392e4 INFO: 172.20.0.1:59164 - "POST /node/69e4383435519646173392e4 HTTP/1.0" 202 Accepted 2026-04-21 02:04:55,984 - lava_callback - INFO - Starting async job submit for node 69e4383435519646173392e4 2026-04-21 02:04:55,984 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:04:55,999 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4383435519646173392e4 HTTP/1.1" 200 None 2026-04-21 02:04:55,999 - lava_callback - DEBUG - Processing node 69e4383435519646173392e4: baseline-arm64-clabbe 2026-04-21 02:04:56,000 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4383435519646173392e4 2026-04-21 02:04:56,000 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:04:56,014 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4383435519646173392e4 HTTP/1.1" 200 None 2026-04-21 02:04:56,016 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:04:56,028 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc360e5dbfa74e32f96891 HTTP/1.1" 200 None 2026-04-21 02:04:56,029 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:04:56,046 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4383435519646173392e4 HTTP/1.1" 200 None 2026-04-21 02:04:56,047 - lava_callback - ERROR - Error processing callback for node 69e4383435519646173392e4: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:59178 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57970 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51892 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51904 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58332 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58342 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55670 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56558 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:05:53,387 - lava_callback - INFO - Received callback for node 69e6db5060f5826562c164fd INFO: 172.20.0.1:48250 - "POST /node/69e6db5060f5826562c164fd HTTP/1.0" 202 Accepted 2026-04-21 02:05:53,398 - lava_callback - INFO - Starting async job submit for node 69e6db5060f5826562c164fd 2026-04-21 02:05:53,442 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:05:53,457 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6db5060f5826562c164fd HTTP/1.1" 200 None 2026-04-21 02:05:53,458 - lava_callback - DEBUG - Processing node 69e6db5060f5826562c164fd: baseline-arm64 2026-04-21 02:05:53,613 - lava_callback - WARNING - Failed to upload log 2026-04-21 02:05:53,689 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:05:53,865 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:05:53,867 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6db5060f5826562c164fd/lava-logs.yaml 2026-04-21 02:05:53,870 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:05:54,143 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:05:54,144 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6db5060f5826562c164fd/lava_callback.json.gz 2026-04-21 02:05:54,175 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:05:54,189 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6db5060f5826562c164fd HTTP/1.1" 200 None 2026-04-21 02:05:54,190 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:05:54,205 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 02:05:54,206 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:05:54,224 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6db5060f5826562c164fd HTTP/1.1" 200 None 2026-04-21 02:05:54,284 - lava_callback - INFO - Completed processing callback for node 69e6db5060f5826562c164fd INFO: 172.20.0.1:48266 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43330 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41978 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 02:06:15,325 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:06:15,340 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-04-21 02:06:16,046 - lava_callback - INFO - Received callback for node 69e4388135519646173392fb INFO: 172.20.0.1:41986 - "POST /node/69e4388135519646173392fb HTTP/1.0" 202 Accepted 2026-04-21 02:06:16,052 - lava_callback - INFO - Starting async job submit for node 69e4388135519646173392fb 2026-04-21 02:06:16,053 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:06:16,067 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4388135519646173392fb HTTP/1.1" 200 None 2026-04-21 02:06:16,068 - lava_callback - DEBUG - Processing node 69e4388135519646173392fb: baseline-arm64-clabbe 2026-04-21 02:06:16,068 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4388135519646173392fb 2026-04-21 02:06:16,069 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:06:16,082 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4388135519646173392fb HTTP/1.1" 200 None 2026-04-21 02:06:16,083 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:06:16,097 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc360e5dbfa74e32f96891 HTTP/1.1" 200 None 2026-04-21 02:06:16,099 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:06:16,116 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4388135519646173392fb HTTP/1.1" 200 None 2026-04-21 02:06:16,117 - lava_callback - ERROR - Error processing callback for node 69e4388135519646173392fb: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:41992 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35818 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35824 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38126 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53906 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42574 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44324 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44564 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 02:07:16,112 - lava_callback - INFO - Received callback for node 69e438ce355196461733931f INFO: 172.20.0.1:44572 - "POST /node/69e438ce355196461733931f HTTP/1.0" 202 Accepted 2026-04-21 02:07:16,117 - lava_callback - INFO - Starting async job submit for node 69e438ce355196461733931f 2026-04-21 02:07:16,118 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:07:16,133 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e438ce355196461733931f HTTP/1.1" 200 None 2026-04-21 02:07:16,134 - lava_callback - DEBUG - Processing node 69e438ce355196461733931f: baseline-arm64-clabbe 2026-04-21 02:07:16,134 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e438ce355196461733931f 2026-04-21 02:07:16,135 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:07:16,148 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e438ce355196461733931f HTTP/1.1" 200 None 2026-04-21 02:07:16,149 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:07:16,162 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69dc360e5dbfa74e32f96891 HTTP/1.1" 200 None 2026-04-21 02:07:16,164 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:07:16,188 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e438ce355196461733931f HTTP/1.1" 200 None 2026-04-21 02:07:16,189 - lava_callback - ERROR - Error processing callback for node 69e438ce355196461733931f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:44574 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44884 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44894 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54072 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40168 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45004 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44936 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39526 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39532 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37124 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37130 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:08:29,437 - lava_callback - INFO - Received callback for node 69e439133551964617339338 INFO: 172.20.0.1:37142 - "POST /node/69e439133551964617339338 HTTP/1.0" 202 Accepted 2026-04-21 02:08:29,442 - lava_callback - INFO - Starting async job submit for node 69e439133551964617339338 2026-04-21 02:08:29,444 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:08:29,461 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e439133551964617339338 HTTP/1.1" 200 None 2026-04-21 02:08:29,462 - lava_callback - DEBUG - Processing node 69e439133551964617339338: baseline-arm64-baylibre 2026-04-21 02:08:29,462 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e439133551964617339338 2026-04-21 02:08:29,463 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:08:29,476 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e439133551964617339338 HTTP/1.1" 200 None 2026-04-21 02:08:29,477 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:08:29,490 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 02:08:29,491 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:08:29,509 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e439133551964617339338 HTTP/1.1" 200 None 2026-04-21 02:08:29,510 - lava_callback - ERROR - Error processing callback for node 69e439133551964617339338: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:46744 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41412 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38548 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44552 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48806 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48812 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49010 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:09:25,314 - lava_callback - INFO - Received callback for node 69e6dbf660f5826562c16540 INFO: 172.20.0.1:49012 - "POST /node/69e6dbf660f5826562c16540 HTTP/1.0" 202 Accepted 2026-04-21 02:09:25,321 - lava_callback - INFO - Starting async job submit for node 69e6dbf660f5826562c16540 2026-04-21 02:09:25,352 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:09:25,366 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6dbf660f5826562c16540 HTTP/1.1" 200 None 2026-04-21 02:09:25,367 - lava_callback - DEBUG - Processing node 69e6dbf660f5826562c16540: baseline-arm64 2026-04-21 02:09:25,561 - lava_callback - WARNING - Failed to upload log 2026-04-21 02:09:25,658 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:09:25,977 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:09:25,979 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6dbf660f5826562c16540/lava-logs.yaml 2026-04-21 02:09:25,984 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:09:26,106 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:09:26,108 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6dbf660f5826562c16540/lava_callback.json.gz 2026-04-21 02:09:26,141 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:09:26,156 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6dbf660f5826562c16540 HTTP/1.1" 200 None 2026-04-21 02:09:26,158 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:09:26,172 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 02:09:26,174 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:09:26,193 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6dbf660f5826562c16540 HTTP/1.1" 200 None 2026-04-21 02:09:26,281 - lava_callback - INFO - Completed processing callback for node 69e6dbf660f5826562c16540 INFO: 172.20.0.1:49028 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57140 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:09:45,342 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:09:45,377 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:40290 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:09:49,570 - lava_callback - INFO - Received callback for node 69e439603551964617339354 INFO: 172.20.0.1:40298 - "POST /node/69e439603551964617339354 HTTP/1.0" 202 Accepted 2026-04-21 02:09:49,577 - lava_callback - INFO - Starting async job submit for node 69e439603551964617339354 2026-04-21 02:09:49,578 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:09:49,597 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e439603551964617339354 HTTP/1.1" 200 None 2026-04-21 02:09:49,598 - lava_callback - DEBUG - Processing node 69e439603551964617339354: baseline-arm64-baylibre 2026-04-21 02:09:49,598 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e439603551964617339354 2026-04-21 02:09:49,599 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:09:49,614 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e439603551964617339354 HTTP/1.1" 200 None 2026-04-21 02:09:49,616 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:09:49,632 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 02:09:49,634 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:09:49,653 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e439603551964617339354 HTTP/1.1" 200 None 2026-04-21 02:09:49,655 - lava_callback - ERROR - Error processing callback for node 69e439603551964617339354: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:32922 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48164 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50402 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50416 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54058 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54072 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44674 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49482 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52112 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51218 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:11:09,617 - lava_callback - INFO - Received callback for node 69e439ac3551964617339373 INFO: 172.20.0.1:51224 - "POST /node/69e439ac3551964617339373 HTTP/1.0" 202 Accepted 2026-04-21 02:11:09,623 - lava_callback - INFO - Starting async job submit for node 69e439ac3551964617339373 2026-04-21 02:11:09,624 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:11:09,637 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e439ac3551964617339373 HTTP/1.1" 200 None 2026-04-21 02:11:09,638 - lava_callback - DEBUG - Processing node 69e439ac3551964617339373: baseline-arm64-baylibre 2026-04-21 02:11:09,638 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e439ac3551964617339373 2026-04-21 02:11:09,639 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:11:09,652 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e439ac3551964617339373 HTTP/1.1" 200 None 2026-04-21 02:11:09,653 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:11:09,665 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 02:11:09,667 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:11:09,688 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e439ac3551964617339373 HTTP/1.1" 200 None 2026-04-21 02:11:09,689 - lava_callback - ERROR - Error processing callback for node 69e439ac3551964617339373: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:48962 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48978 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41856 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41872 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32850 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35824 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:11:55,501 - lava_callback - INFO - Received callback for node 69e6da5b60f5826562c164a2 INFO: 172.20.0.1:53726 - "POST /node/69e6da5b60f5826562c164a2 HTTP/1.0" 202 Accepted 2026-04-21 02:11:55,508 - lava_callback - INFO - Starting async job submit for node 69e6da5b60f5826562c164a2 2026-04-21 02:11:55,538 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:11:55,561 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6da5b60f5826562c164a2 HTTP/1.1" 200 None 2026-04-21 02:11:55,562 - lava_callback - DEBUG - Processing node 69e6da5b60f5826562c164a2: baseline-arm64 2026-04-21 02:11:56,139 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:11:56,259 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:11:56,261 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6da5b60f5826562c164a2/log.txt.gz INFO: 172.20.0.1:53730 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:11:56,521 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:11:56,692 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:11:56,694 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6da5b60f5826562c164a2/lava-logs.yaml 2026-04-21 02:11:56,706 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:11:56,820 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:11:56,822 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6da5b60f5826562c164a2/lava_callback.json.gz 2026-04-21 02:11:56,847 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:11:56,861 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6da5b60f5826562c164a2 HTTP/1.1" 200 None 2026-04-21 02:11:56,862 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:11:56,938 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 02:11:56,940 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:11:56,962 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6da5b60f5826562c164a2 HTTP/1.1" 200 None 2026-04-21 02:11:57,024 - lava_callback - INFO - Completed processing callback for node 69e6da5b60f5826562c164a2 INFO: 172.20.0.1:33256 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48602 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 02:12:15,379 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:12:15,401 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:48618 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34198 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34204 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:12:29,692 - lava_callback - INFO - Received callback for node 69e439fa3551964617339395 INFO: 172.20.0.1:34214 - "POST /node/69e439fa3551964617339395 HTTP/1.0" 202 Accepted 2026-04-21 02:12:29,698 - lava_callback - INFO - Starting async job submit for node 69e439fa3551964617339395 2026-04-21 02:12:29,699 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:12:29,713 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e439fa3551964617339395 HTTP/1.1" 200 None 2026-04-21 02:12:29,714 - lava_callback - DEBUG - Processing node 69e439fa3551964617339395: baseline-arm64-baylibre 2026-04-21 02:12:29,714 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e439fa3551964617339395 2026-04-21 02:12:29,715 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:12:29,728 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e439fa3551964617339395 HTTP/1.1" 200 None 2026-04-21 02:12:29,729 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:12:29,742 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 02:12:29,743 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:12:29,762 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e439fa3551964617339395 HTTP/1.1" 200 None 2026-04-21 02:12:29,763 - lava_callback - ERROR - Error processing callback for node 69e439fa3551964617339395: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:33060 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47858 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36436 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43216 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48520 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48530 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57348 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57362 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:13:29,742 - lava_callback - INFO - Received callback for node 69e43a4635519646173393b1 INFO: 172.20.0.1:49672 - "POST /node/69e43a4635519646173393b1 HTTP/1.0" 202 Accepted 2026-04-21 02:13:29,747 - lava_callback - INFO - Starting async job submit for node 69e43a4635519646173393b1 2026-04-21 02:13:29,748 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:13:29,763 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43a4635519646173393b1 HTTP/1.1" 200 None 2026-04-21 02:13:29,764 - lava_callback - DEBUG - Processing node 69e43a4635519646173393b1: baseline-arm64-baylibre 2026-04-21 02:13:29,764 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e43a4635519646173393b1 2026-04-21 02:13:29,765 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:13:29,778 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43a4635519646173393b1 HTTP/1.1" 200 None 2026-04-21 02:13:29,779 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:13:29,792 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 02:13:29,793 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:13:29,821 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e43a4635519646173393b1 HTTP/1.1" 200 None 2026-04-21 02:13:29,822 - lava_callback - ERROR - Error processing callback for node 69e43a4635519646173393b1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49684 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43126 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54362 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48114 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41104 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41114 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48534 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48550 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49118 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60696 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:14:49,777 - lava_callback - INFO - Received callback for node 69e43a9235519646173393d3 INFO: 172.20.0.1:59010 - "POST /node/69e43a9235519646173393d3 HTTP/1.0" 202 Accepted 2026-04-21 02:14:49,783 - lava_callback - INFO - Starting async job submit for node 69e43a9235519646173393d3 2026-04-21 02:14:49,784 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:14:49,802 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43a9235519646173393d3 HTTP/1.1" 200 None 2026-04-21 02:14:49,803 - lava_callback - DEBUG - Processing node 69e43a9235519646173393d3: baseline-arm64-baylibre 2026-04-21 02:14:49,803 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e43a9235519646173393d3 2026-04-21 02:14:49,804 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:14:49,817 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43a9235519646173393d3 HTTP/1.1" 200 None 2026-04-21 02:14:49,818 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:14:49,831 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 02:14:49,833 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:14:49,849 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e43a9235519646173393d3 HTTP/1.1" 200 None 2026-04-21 02:14:49,850 - lava_callback - ERROR - Error processing callback for node 69e43a9235519646173393d3: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:59014 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58318 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38628 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:38638 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35426 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35436 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39148 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40710 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45078 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32872 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55794 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 02:16:16,456 - lava_callback - INFO - Received callback for node 69e43ae435519646173393f1 INFO: 172.20.0.1:55802 - "POST /node/69e43ae435519646173393f1 HTTP/1.0" 202 Accepted 2026-04-21 02:16:16,462 - lava_callback - INFO - Starting async job submit for node 69e43ae435519646173393f1 2026-04-21 02:16:16,463 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:16:16,476 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43ae435519646173393f1 HTTP/1.1" 200 None 2026-04-21 02:16:16,477 - lava_callback - DEBUG - Processing node 69e43ae435519646173393f1: baseline-arm64-clabbe 2026-04-21 02:16:16,477 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e43ae435519646173393f1 2026-04-21 02:16:16,478 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:16:16,491 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43ae435519646173393f1 HTTP/1.1" 200 None 2026-04-21 02:16:16,493 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:16:16,506 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 02:16:16,507 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:16:16,524 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e43ae435519646173393f1 HTTP/1.1" 200 None INFO: 172.20.0.1:55804 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:16:16,527 - lava_callback - ERROR - Error processing callback for node 69e43ae435519646173393f1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:52836 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52842 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45188 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47870 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44648 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45750 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60170 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60176 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47454 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47464 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:17:36,509 - lava_callback - INFO - Received callback for node 69e43b313551964617339412 INFO: 172.20.0.1:59540 - "POST /node/69e43b313551964617339412 HTTP/1.0" 202 Accepted 2026-04-21 02:17:36,515 - lava_callback - INFO - Starting async job submit for node 69e43b313551964617339412 2026-04-21 02:17:36,515 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 INFO: 172.20.0.1:59548 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:17:36,538 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43b313551964617339412 HTTP/1.1" 200 None 2026-04-21 02:17:36,539 - lava_callback - DEBUG - Processing node 69e43b313551964617339412: baseline-arm64-clabbe 2026-04-21 02:17:36,539 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e43b313551964617339412 2026-04-21 02:17:36,540 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:17:36,554 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43b313551964617339412 HTTP/1.1" 200 None 2026-04-21 02:17:36,556 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:17:36,570 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 02:17:36,571 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:17:36,588 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e43b313551964617339412 HTTP/1.1" 200 None 2026-04-21 02:17:36,589 - lava_callback - ERROR - Error processing callback for node 69e43b313551964617339412: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:44274 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53640 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59934 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39074 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39078 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45686 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45698 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57850 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43258 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50062 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:18:56,607 - lava_callback - INFO - Received callback for node 69e43b7d3551964617339437 INFO: 172.20.0.1:50064 - "POST /node/69e43b7d3551964617339437 HTTP/1.0" 202 Accepted 2026-04-21 02:18:56,613 - lava_callback - INFO - Starting async job submit for node 69e43b7d3551964617339437 2026-04-21 02:18:56,614 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:18:56,628 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43b7d3551964617339437 HTTP/1.1" 200 None 2026-04-21 02:18:56,629 - lava_callback - DEBUG - Processing node 69e43b7d3551964617339437: baseline-arm64-clabbe 2026-04-21 02:18:56,629 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e43b7d3551964617339437 2026-04-21 02:18:56,630 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:18:56,642 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43b7d3551964617339437 HTTP/1.1" 200 None 2026-04-21 02:18:56,644 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:18:56,656 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 02:18:56,658 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:18:56,675 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e43b7d3551964617339437 HTTP/1.1" 200 None 2026-04-21 02:18:56,676 - lava_callback - ERROR - Error processing callback for node 69e43b7d3551964617339437: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47310 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54328 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:54332 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50954 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50960 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:19:36,155 - lava_callback - INFO - Received callback for node 69e6da0b60f5826562c16481 INFO: 172.20.0.1:44104 - "POST /node/69e6da0b60f5826562c16481 HTTP/1.0" 202 Accepted 2026-04-21 02:19:36,166 - lava_callback - INFO - Starting async job submit for node 69e6da0b60f5826562c16481 2026-04-21 02:19:36,210 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:19:36,224 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6da0b60f5826562c16481 HTTP/1.1" 200 None 2026-04-21 02:19:36,225 - lava_callback - DEBUG - Processing node 69e6da0b60f5826562c16481: baseline-nfs-arm64 INFO: 172.20.0.1:44110 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:19:37,436 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:19:37,756 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:19:37,758 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6da0b60f5826562c16481/log.txt.gz 2026-04-21 02:19:38,393 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:19:38,624 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:19:38,627 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6da0b60f5826562c16481/lava-logs.yaml 2026-04-21 02:19:38,652 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:19:38,842 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:19:38,844 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6da0b60f5826562c16481/lava_callback.json.gz DEBUG: Setup failed for 69e6da0b60f5826562c16481. Transitting job node result: incomplete -> fail 2026-04-21 02:19:38,932 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:19:38,946 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6da0b60f5826562c16481 HTTP/1.1" 200 None 2026-04-21 02:19:38,948 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:19:38,980 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 02:19:38,981 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:19:39,007 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6da0b60f5826562c16481 HTTP/1.1" 200 None 2026-04-21 02:19:39,091 - lava_callback - INFO - Completed processing callback for node 69e6da0b60f5826562c16481 2026-04-21 02:19:45,404 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:19:45,420 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:46022 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39266 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:19:56,639 - lava_callback - INFO - Received callback for node 69e43bca3551964617339450 INFO: 172.20.0.1:39270 - "POST /node/69e43bca3551964617339450 HTTP/1.0" 202 Accepted 2026-04-21 02:19:56,646 - lava_callback - INFO - Starting async job submit for node 69e43bca3551964617339450 2026-04-21 02:19:56,647 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:19:56,664 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43bca3551964617339450 HTTP/1.1" 200 None 2026-04-21 02:19:56,665 - lava_callback - DEBUG - Processing node 69e43bca3551964617339450: baseline-arm64-clabbe 2026-04-21 02:19:56,665 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e43bca3551964617339450 2026-04-21 02:19:56,666 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:19:56,681 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43bca3551964617339450 HTTP/1.1" 200 None 2026-04-21 02:19:56,682 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:19:56,694 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 02:19:56,696 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:19:56,721 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e43bca3551964617339450 HTTP/1.1" 200 None 2026-04-21 02:19:56,722 - lava_callback - ERROR - Error processing callback for node 69e43bca3551964617339450: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:50908 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51634 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51640 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34454 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34456 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38738 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42800 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43048 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34216 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34714 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34716 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:21:16,636 - lava_callback - INFO - Received callback for node 69e43c153551964617339478 INFO: 172.20.0.1:34720 - "POST /node/69e43c153551964617339478 HTTP/1.0" 202 Accepted 2026-04-21 02:21:16,641 - lava_callback - INFO - Starting async job submit for node 69e43c153551964617339478 2026-04-21 02:21:16,642 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:21:16,657 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43c153551964617339478 HTTP/1.1" 200 None 2026-04-21 02:21:16,658 - lava_callback - DEBUG - Processing node 69e43c153551964617339478: baseline-arm64-clabbe 2026-04-21 02:21:16,658 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e43c153551964617339478 2026-04-21 02:21:16,659 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:21:16,672 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43c153551964617339478 HTTP/1.1" 200 None 2026-04-21 02:21:16,673 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:21:16,686 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 02:21:16,687 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:21:16,705 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e43c153551964617339478 HTTP/1.1" 200 None 2026-04-21 02:21:16,706 - lava_callback - ERROR - Error processing callback for node 69e43c153551964617339478: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:40394 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40404 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60060 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48606 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57986 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33232 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55074 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:55090 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35330 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35336 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:22:30,630 - lava_callback - INFO - Received callback for node 69e6dab360f5826562c164c1 INFO: 172.20.0.1:37414 - "POST /node/69e6dab360f5826562c164c1 HTTP/1.0" 202 Accepted 2026-04-21 02:22:30,638 - lava_callback - INFO - Starting async job submit for node 69e6dab360f5826562c164c1 2026-04-21 02:22:30,686 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:22:30,700 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6dab360f5826562c164c1 HTTP/1.1" 200 None 2026-04-21 02:22:30,701 - lava_callback - DEBUG - Processing node 69e6dab360f5826562c164c1: baseline-nfs-arm64 2026-04-21 02:22:31,854 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:22:32,010 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:22:32,012 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6dab360f5826562c164c1/log.txt.gz 2026-04-21 02:22:32,707 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:22:32,915 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:22:32,918 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6dab360f5826562c164c1/lava-logs.yaml 2026-04-21 02:22:32,940 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:22:33,085 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:22:33,086 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6dab360f5826562c164c1/lava_callback.json.gz DEBUG: Setup failed for 69e6dab360f5826562c164c1. Transitting job node result: incomplete -> fail 2026-04-21 02:22:33,174 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:22:33,188 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6dab360f5826562c164c1 HTTP/1.1" 200 None 2026-04-21 02:22:33,190 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:22:33,203 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 02:22:33,204 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:22:33,225 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6dab360f5826562c164c1 HTTP/1.1" 200 None 2026-04-21 02:22:33,311 - lava_callback - INFO - Completed processing callback for node 69e6dab360f5826562c164c1 INFO: 172.20.0.1:37420 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:22:36,849 - lava_callback - INFO - Received callback for node 69e43c603551964617339493 INFO: 172.20.0.1:37436 - "POST /node/69e43c603551964617339493 HTTP/1.0" 202 Accepted 2026-04-21 02:22:36,855 - lava_callback - INFO - Starting async job submit for node 69e43c603551964617339493 2026-04-21 02:22:36,856 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:22:36,896 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43c603551964617339493 HTTP/1.1" 200 None 2026-04-21 02:22:36,897 - lava_callback - DEBUG - Processing node 69e43c603551964617339493: baseline-arm64-clabbe 2026-04-21 02:22:36,897 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e43c603551964617339493 2026-04-21 02:22:36,898 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:22:36,936 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43c603551964617339493 HTTP/1.1" 200 None 2026-04-21 02:22:36,938 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:22:36,959 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 02:22:36,961 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:22:36,979 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e43c603551964617339493 HTTP/1.1" 200 None 2026-04-21 02:22:36,981 - lava_callback - ERROR - Error processing callback for node 69e43c603551964617339493: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-04-21 02:22:45,422 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:22:45,444 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:37546 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43134 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45706 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43282 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43284 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36560 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36570 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36494 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59024 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58064 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:23:56,772 - lava_callback - INFO - Received callback for node 69e43cab35519646173394b7 INFO: 172.20.0.1:58066 - "POST /node/69e43cab35519646173394b7 HTTP/1.0" 202 Accepted 2026-04-21 02:23:56,778 - lava_callback - INFO - Starting async job submit for node 69e43cab35519646173394b7 2026-04-21 02:23:56,779 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:23:56,793 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43cab35519646173394b7 HTTP/1.1" 200 None 2026-04-21 02:23:56,794 - lava_callback - DEBUG - Processing node 69e43cab35519646173394b7: baseline-arm64-clabbe 2026-04-21 02:23:56,794 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e43cab35519646173394b7 2026-04-21 02:23:56,794 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:23:56,812 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43cab35519646173394b7 HTTP/1.1" 200 None 2026-04-21 02:23:56,814 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:23:56,827 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de68bcf6535ffd5b3ed8c4 HTTP/1.1" 200 None 2026-04-21 02:23:56,828 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:23:56,845 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e43cab35519646173394b7 HTTP/1.1" 200 None 2026-04-21 02:23:56,846 - lava_callback - ERROR - Error processing callback for node 69e43cab35519646173394b7: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:35220 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48846 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48858 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39800 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:24:23,304 - lava_callback - INFO - Received callback for node 69e6daff60f5826562c164de INFO: 172.20.0.1:39810 - "POST /node/69e6daff60f5826562c164de HTTP/1.0" 202 Accepted 2026-04-21 02:24:23,310 - lava_callback - INFO - Starting async job submit for node 69e6daff60f5826562c164de 2026-04-21 02:24:23,338 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:24:23,351 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6daff60f5826562c164de HTTP/1.1" 200 None 2026-04-21 02:24:23,352 - lava_callback - DEBUG - Processing node 69e6daff60f5826562c164de: baseline-arm64 2026-04-21 02:24:23,652 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:24:23,764 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:24:23,766 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6daff60f5826562c164de/log.txt.gz 2026-04-21 02:24:23,877 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:24:24,195 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:24:24,197 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6daff60f5826562c164de/lava-logs.yaml 2026-04-21 02:24:24,203 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:24:24,320 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:24:24,322 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6daff60f5826562c164de/lava_callback.json.gz DEBUG: Setup failed for 69e6daff60f5826562c164de. Transitting job node result: incomplete -> fail 2026-04-21 02:24:24,364 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:24:24,379 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6daff60f5826562c164de HTTP/1.1" 200 None 2026-04-21 02:24:24,380 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:24:24,393 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 02:24:24,395 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:24:24,415 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6daff60f5826562c164de HTTP/1.1" 200 None 2026-04-21 02:24:24,463 - lava_callback - INFO - Completed processing callback for node 69e6daff60f5826562c164de INFO: 172.20.0.1:39812 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35690 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:24:45,446 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:24:45,490 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:34190 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:24:50,379 - lava_callback - INFO - Received callback for node 69e43ced35519646173394d4 INFO: 172.20.0.1:49660 - "POST /node/69e43ced35519646173394d4 HTTP/1.0" 202 Accepted 2026-04-21 02:24:50,385 - lava_callback - INFO - Starting async job submit for node 69e43ced35519646173394d4 2026-04-21 02:24:50,385 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:24:50,399 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43ced35519646173394d4 HTTP/1.1" 200 None 2026-04-21 02:24:50,400 - lava_callback - DEBUG - Processing node 69e43ced35519646173394d4: baseline-arm64-baylibre 2026-04-21 02:24:50,400 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e43ced35519646173394d4 2026-04-21 02:24:50,401 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:24:50,414 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43ced35519646173394d4 HTTP/1.1" 200 None 2026-04-21 02:24:50,415 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:24:50,430 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 02:24:50,432 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:24:50,450 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e43ced35519646173394d4 HTTP/1.1" 200 None 2026-04-21 02:24:50,452 - lava_callback - ERROR - Error processing callback for node 69e43ced35519646173394d4: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49676 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40410 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55320 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:55322 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36774 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36778 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50574 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32962 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55876 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37390 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:26:10,499 - lava_callback - INFO - Received callback for node 69e43d3a35519646173394f3 INFO: 172.20.0.1:54116 - "POST /node/69e43d3a35519646173394f3 HTTP/1.0" 202 Accepted 2026-04-21 02:26:10,504 - lava_callback - INFO - Starting async job submit for node 69e43d3a35519646173394f3 2026-04-21 02:26:10,505 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:26:10,519 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43d3a35519646173394f3 HTTP/1.1" 200 None 2026-04-21 02:26:10,520 - lava_callback - DEBUG - Processing node 69e43d3a35519646173394f3: baseline-arm64-baylibre 2026-04-21 02:26:10,520 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e43d3a35519646173394f3 2026-04-21 02:26:10,521 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:26:10,534 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43d3a35519646173394f3 HTTP/1.1" 200 None 2026-04-21 02:26:10,535 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:26:10,548 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 02:26:10,550 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:26:10,568 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e43d3a35519646173394f3 HTTP/1.1" 200 None 2026-04-21 02:26:10,568 - lava_callback - ERROR - Error processing callback for node 69e43d3a35519646173394f3: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54126 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:54138 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52836 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52852 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39544 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45340 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46840 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34870 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50662 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50672 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56694 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:27:25,393 - lava_callback - INFO - Received callback for node 69e6dc9e60f5826562c1659c INFO: 172.20.0.1:56702 - "POST /node/69e6dc9e60f5826562c1659c HTTP/1.0" 202 Accepted 2026-04-21 02:27:25,401 - lava_callback - INFO - Starting async job submit for node 69e6dc9e60f5826562c1659c 2026-04-21 02:27:25,457 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:27:25,471 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6dc9e60f5826562c1659c HTTP/1.1" 200 None 2026-04-21 02:27:25,472 - lava_callback - DEBUG - Processing node 69e6dc9e60f5826562c1659c: baseline-arm 2026-04-21 02:27:26,060 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:27:26,367 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:27:26,369 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69e6dc9e60f5826562c1659c/log.txt.gz INFO: 172.20.0.1:56716 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:27:26,745 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:27:26,937 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:27:26,939 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69e6dc9e60f5826562c1659c/lava-logs.yaml 2026-04-21 02:27:26,954 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:27:27,058 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:27:27,059 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69e6dc9e60f5826562c1659c/lava_callback.json.gz DEBUG: Setup failed for 69e6dc9e60f5826562c1659c. Transitting job node result: incomplete -> fail 2026-04-21 02:27:27,167 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:27:27,182 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6dc9e60f5826562c1659c HTTP/1.1" 200 None 2026-04-21 02:27:27,183 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:27:27,196 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 02:27:27,198 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:27:27,238 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6dc9e60f5826562c1659c HTTP/1.1" 200 None 2026-04-21 02:27:27,339 - lava_callback - INFO - Completed processing callback for node 69e6dc9e60f5826562c1659c 2026-04-21 02:27:30,534 - lava_callback - INFO - Received callback for node 69e43d873551964617339513 INFO: 172.20.0.1:40306 - "POST /node/69e43d873551964617339513 HTTP/1.0" 202 Accepted 2026-04-21 02:27:30,543 - lava_callback - INFO - Starting async job submit for node 69e43d873551964617339513 2026-04-21 02:27:30,544 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:27:30,561 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43d873551964617339513 HTTP/1.1" 200 None 2026-04-21 02:27:30,562 - lava_callback - DEBUG - Processing node 69e43d873551964617339513: baseline-arm64-baylibre 2026-04-21 02:27:30,562 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e43d873551964617339513 2026-04-21 02:27:30,563 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:27:30,601 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43d873551964617339513 HTTP/1.1" 200 None 2026-04-21 02:27:30,603 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:27:30,629 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 02:27:30,631 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:27:30,650 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e43d873551964617339513 HTTP/1.1" 200 None 2026-04-21 02:27:30,652 - lava_callback - ERROR - Error processing callback for node 69e43d873551964617339513: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-04-21 02:27:33,638 - lava_callback - INFO - Received callback for node 69e6dbaa60f5826562c16521 INFO: 172.20.0.1:40310 - "POST /node/69e6dbaa60f5826562c16521 HTTP/1.0" 202 Accepted 2026-04-21 02:27:33,644 - lava_callback - INFO - Starting async job submit for node 69e6dbaa60f5826562c16521 2026-04-21 02:27:33,666 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:27:33,683 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6dbaa60f5826562c16521 HTTP/1.1" 200 None 2026-04-21 02:27:33,684 - lava_callback - DEBUG - Processing node 69e6dbaa60f5826562c16521: baseline-nfs-arm64 2026-04-21 02:27:33,911 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:27:34,013 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:27:34,015 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6dbaa60f5826562c16521/log.txt.gz 2026-04-21 02:27:34,139 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:27:34,263 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:27:34,265 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6dbaa60f5826562c16521/lava-logs.yaml 2026-04-21 02:27:34,272 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:27:34,382 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:27:34,383 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6dbaa60f5826562c16521/lava_callback.json.gz DEBUG: Setup failed for 69e6dbaa60f5826562c16521. Transitting job node result: incomplete -> fail 2026-04-21 02:27:34,426 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:27:34,439 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6dbaa60f5826562c16521 HTTP/1.1" 200 None 2026-04-21 02:27:34,440 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:27:34,482 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 02:27:34,484 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:27:34,503 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6dbaa60f5826562c16521 HTTP/1.1" 200 None 2026-04-21 02:27:34,537 - lava_callback - INFO - Completed processing callback for node 69e6dbaa60f5826562c16521 INFO: 172.20.0.1:40318 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:27:45,493 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:27:45,533 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:41934 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39712 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36336 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46300 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46314 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49120 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49134 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40808 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:28:46,243 - lava_callback - INFO - Received callback for node 69e6dcc260f5826562c165a8 INFO: 172.20.0.1:46566 - "POST /node/69e6dcc260f5826562c165a8 HTTP/1.0" 202 Accepted 2026-04-21 02:28:46,250 - lava_callback - INFO - Starting async job submit for node 69e6dcc260f5826562c165a8 2026-04-21 02:28:46,308 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:28:46,325 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6dcc260f5826562c165a8 HTTP/1.1" 200 None 2026-04-21 02:28:46,326 - lava_callback - DEBUG - Processing node 69e6dcc260f5826562c165a8: ltp-smoke INFO: 172.20.0.1:46580 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:28:46,841 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:28:46,973 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:28:46,975 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-smoke-69e6dcc260f5826562c165a8/log.txt.gz 2026-04-21 02:28:47,241 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:28:47,425 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:28:47,427 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-smoke-69e6dcc260f5826562c165a8/lava-logs.yaml 2026-04-21 02:28:47,441 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:28:47,566 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:28:47,568 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-smoke-69e6dcc260f5826562c165a8/lava_callback.json.gz 2026-04-21 02:28:47,687 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:28:47,701 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6dcc260f5826562c165a8 HTTP/1.1" 200 None 2026-04-21 02:28:47,703 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:28:47,715 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c21560f5826562c15091 HTTP/1.1" 200 None 2026-04-21 02:28:47,717 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:28:47,737 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6dcc260f5826562c165a8 HTTP/1.1" 200 None 2026-04-21 02:28:47,843 - lava_callback - INFO - Completed processing callback for node 69e6dcc260f5826562c165a8 2026-04-21 02:28:50,580 - lava_callback - INFO - Received callback for node 69e43dd7355196461733953a INFO: 172.20.0.1:49400 - "POST /node/69e43dd7355196461733953a HTTP/1.0" 202 Accepted 2026-04-21 02:28:50,585 - lava_callback - INFO - Starting async job submit for node 69e43dd7355196461733953a 2026-04-21 02:28:50,586 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:28:50,603 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43dd7355196461733953a HTTP/1.1" 200 None 2026-04-21 02:28:50,603 - lava_callback - DEBUG - Processing node 69e43dd7355196461733953a: baseline-arm64-baylibre 2026-04-21 02:28:50,604 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e43dd7355196461733953a 2026-04-21 02:28:50,604 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:28:50,617 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43dd7355196461733953a HTTP/1.1" 200 None 2026-04-21 02:28:50,618 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:28:50,631 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 02:28:50,632 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:28:50,651 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e43dd7355196461733953a HTTP/1.1" 200 None 2026-04-21 02:28:50,653 - lava_callback - ERROR - Error processing callback for node 69e43dd7355196461733953a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49410 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37416 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60342 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 02:29:15,536 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:29:15,553 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:60344 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49918 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49928 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59810 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53628 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40612 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44612 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:30:10,681 - lava_callback - INFO - Received callback for node 69e43e23355196461733955b INFO: 172.20.0.1:37314 - "POST /node/69e43e23355196461733955b HTTP/1.0" 202 Accepted 2026-04-21 02:30:10,687 - lava_callback - INFO - Starting async job submit for node 69e43e23355196461733955b 2026-04-21 02:30:10,688 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:30:10,711 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43e23355196461733955b HTTP/1.1" 200 None 2026-04-21 02:30:10,712 - lava_callback - DEBUG - Processing node 69e43e23355196461733955b: baseline-arm64-baylibre 2026-04-21 02:30:10,712 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e43e23355196461733955b 2026-04-21 02:30:10,713 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:30:10,726 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43e23355196461733955b HTTP/1.1" 200 None 2026-04-21 02:30:10,727 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:30:10,740 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 02:30:10,741 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:30:10,761 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e43e23355196461733955b HTTP/1.1" 200 None 2026-04-21 02:30:10,762 - lava_callback - ERROR - Error processing callback for node 69e43e23355196461733955b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:37318 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37326 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35918 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35920 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:30:31,420 - lava_callback - INFO - Received callback for node 69e6e01460f5826562c16723 INFO: 172.20.0.1:34314 - "POST /node/69e6e01460f5826562c16723 HTTP/1.0" 202 Accepted 2026-04-21 02:30:31,427 - lava_callback - INFO - Starting async job submit for node 69e6e01460f5826562c16723 2026-04-21 02:30:31,464 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:30:31,477 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6e01460f5826562c16723 HTTP/1.1" 200 None 2026-04-21 02:30:31,478 - lava_callback - DEBUG - Processing node 69e6e01460f5826562c16723: baseline-arm64 2026-04-21 02:30:31,869 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:30:32,090 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:30:32,091 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6e01460f5826562c16723/log.txt.gz 2026-04-21 02:30:32,299 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:30:32,654 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:30:32,656 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6e01460f5826562c16723/lava-logs.yaml 2026-04-21 02:30:32,667 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:30:32,812 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:30:32,814 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6e01460f5826562c16723/lava_callback.json.gz DEBUG: Setup failed for 69e6e01460f5826562c16723. Transitting job node result: incomplete -> fail 2026-04-21 02:30:32,893 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:30:32,908 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6e01460f5826562c16723 HTTP/1.1" 200 None 2026-04-21 02:30:32,910 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:30:32,924 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 02:30:32,926 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:30:32,946 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6e01460f5826562c16723 HTTP/1.1" 200 None 2026-04-21 02:30:33,014 - lava_callback - INFO - Completed processing callback for node 69e6e01460f5826562c16723 INFO: 172.20.0.1:34324 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:30:45,555 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:30:45,582 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:60118 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40516 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48940 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42440 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:42448 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40462 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40464 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:31:30,748 - lava_callback - INFO - Received callback for node 69e43e79355196461733957d INFO: 172.20.0.1:37248 - "POST /node/69e43e79355196461733957d HTTP/1.0" 202 Accepted 2026-04-21 02:31:30,754 - lava_callback - INFO - Starting async job submit for node 69e43e79355196461733957d 2026-04-21 02:31:30,755 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:31:30,769 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43e79355196461733957d HTTP/1.1" 200 None 2026-04-21 02:31:30,770 - lava_callback - DEBUG - Processing node 69e43e79355196461733957d: baseline-arm64-baylibre 2026-04-21 02:31:30,770 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e43e79355196461733957d 2026-04-21 02:31:30,771 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:31:30,784 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43e79355196461733957d HTTP/1.1" 200 None 2026-04-21 02:31:30,785 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:31:30,798 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 02:31:30,800 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:31:30,816 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e43e79355196461733957d HTTP/1.1" 200 None 2026-04-21 02:31:30,817 - lava_callback - ERROR - Error processing callback for node 69e43e79355196461733957d: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:37260 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56228 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42592 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44244 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48372 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48382 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37102 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37118 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36778 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40856 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51436 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:32:57,148 - lava_callback - INFO - Received callback for node 69e43ecf35519646173395a6 INFO: 172.20.0.1:51440 - "POST /node/69e43ecf35519646173395a6 HTTP/1.0" 202 Accepted 2026-04-21 02:32:57,154 - lava_callback - INFO - Starting async job submit for node 69e43ecf35519646173395a6 2026-04-21 02:32:57,155 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:32:57,182 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43ecf35519646173395a6 HTTP/1.1" 200 None 2026-04-21 02:32:57,183 - lava_callback - DEBUG - Processing node 69e43ecf35519646173395a6: baseline-arm64-clabbe 2026-04-21 02:32:57,183 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e43ecf35519646173395a6 2026-04-21 02:32:57,184 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:32:57,197 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43ecf35519646173395a6 HTTP/1.1" 200 None 2026-04-21 02:32:57,198 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:32:57,211 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 02:32:57,212 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:32:57,234 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e43ecf35519646173395a6 HTTP/1.1" 200 None 2026-04-21 02:32:57,235 - lava_callback - ERROR - Error processing callback for node 69e43ecf35519646173395a6: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:57380 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47070 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47072 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37486 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37488 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60250 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33152 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36598 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38072 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33100 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33108 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:34:17,158 - lava_callback - INFO - Received callback for node 69e43f1b35519646173395c3 INFO: 172.20.0.1:33114 - "POST /node/69e43f1b35519646173395c3 HTTP/1.0" 202 Accepted 2026-04-21 02:34:17,164 - lava_callback - INFO - Starting async job submit for node 69e43f1b35519646173395c3 2026-04-21 02:34:17,165 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:34:17,181 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43f1b35519646173395c3 HTTP/1.1" 200 None 2026-04-21 02:34:17,182 - lava_callback - DEBUG - Processing node 69e43f1b35519646173395c3: baseline-arm64-clabbe 2026-04-21 02:34:17,183 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e43f1b35519646173395c3 2026-04-21 02:34:17,184 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:34:17,198 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43f1b35519646173395c3 HTTP/1.1" 200 None 2026-04-21 02:34:17,200 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:34:17,214 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded948f490930782fb6e2d HTTP/1.1" 200 None 2026-04-21 02:34:17,215 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:34:17,233 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e43f1b35519646173395c3 HTTP/1.1" 200 None 2026-04-21 02:34:17,235 - lava_callback - ERROR - Error processing callback for node 69e43f1b35519646173395c3: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:58924 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58940 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56738 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42852 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43744 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53268 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50444 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50458 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49826 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49828 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51100 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:35:37,235 - lava_callback - INFO - Received callback for node 69e43f6735519646173395e5 INFO: 172.20.0.1:51108 - "POST /node/69e43f6735519646173395e5 HTTP/1.0" 202 Accepted 2026-04-21 02:35:37,241 - lava_callback - INFO - Starting async job submit for node 69e43f6735519646173395e5 2026-04-21 02:35:37,242 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:35:37,256 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43f6735519646173395e5 HTTP/1.1" 200 None 2026-04-21 02:35:37,257 - lava_callback - DEBUG - Processing node 69e43f6735519646173395e5: baseline-arm64-clabbe 2026-04-21 02:35:37,257 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e43f6735519646173395e5 2026-04-21 02:35:37,258 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:35:37,271 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43f6735519646173395e5 HTTP/1.1" 200 None 2026-04-21 02:35:37,273 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:35:37,285 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 02:35:37,286 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:35:37,305 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e43f6735519646173395e5 HTTP/1.1" 200 None 2026-04-21 02:35:37,306 - lava_callback - ERROR - Error processing callback for node 69e43f6735519646173395e5: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:59554 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59166 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42312 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56390 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:56406 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46618 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46622 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52976 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:36:37,551 - lava_callback - INFO - Received callback for node 69e6dc5160f5826562c16572 INFO: 172.20.0.1:52984 - "POST /node/69e6dc5160f5826562c16572 HTTP/1.0" 202 Accepted 2026-04-21 02:36:37,558 - lava_callback - INFO - Starting async job submit for node 69e6dc5160f5826562c16572 2026-04-21 02:36:37,584 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:36:37,597 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6dc5160f5826562c16572 HTTP/1.1" 200 None 2026-04-21 02:36:37,598 - lava_callback - DEBUG - Processing node 69e6dc5160f5826562c16572: baseline-nfs-arm64 2026-04-21 02:36:38,173 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:36:38,470 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:36:38,473 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6dc5160f5826562c16572/log.txt.gz 2026-04-21 02:36:38,746 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:36:39,081 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:36:39,083 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6dc5160f5826562c16572/lava-logs.yaml 2026-04-21 02:36:39,095 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:36:39,244 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:36:39,246 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6dc5160f5826562c16572/lava_callback.json.gz 2026-04-21 02:36:39,271 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:36:39,285 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6dc5160f5826562c16572 HTTP/1.1" 200 None 2026-04-21 02:36:39,286 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:36:39,308 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 02:36:39,309 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:36:39,327 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6dc5160f5826562c16572 HTTP/1.1" 200 None 2026-04-21 02:36:39,387 - lava_callback - INFO - Completed processing callback for node 69e6dc5160f5826562c16572 2026-04-21 02:36:45,585 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:36:45,614 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:37518 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47278 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:36:57,311 - lava_callback - INFO - Received callback for node 69e43fb33551964617339605 INFO: 172.20.0.1:47292 - "POST /node/69e43fb33551964617339605 HTTP/1.0" 202 Accepted 2026-04-21 02:36:57,317 - lava_callback - INFO - Starting async job submit for node 69e43fb33551964617339605 2026-04-21 02:36:57,318 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:36:57,343 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43fb33551964617339605 HTTP/1.1" 200 None 2026-04-21 02:36:57,344 - lava_callback - DEBUG - Processing node 69e43fb33551964617339605: baseline-arm64-clabbe 2026-04-21 02:36:57,344 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e43fb33551964617339605 2026-04-21 02:36:57,345 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:36:57,358 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43fb33551964617339605 HTTP/1.1" 200 None 2026-04-21 02:36:57,360 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:36:57,375 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 02:36:57,376 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:36:57,394 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e43fb33551964617339605 HTTP/1.1" 200 None 2026-04-21 02:36:57,395 - lava_callback - ERROR - Error processing callback for node 69e43fb33551964617339605: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:39800 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53072 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53076 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51344 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51360 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48080 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36296 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:37:51,077 - lava_callback - INFO - Received callback for node 69e43ff63551964617339626 INFO: 172.20.0.1:56122 - "POST /node/69e43ff63551964617339626 HTTP/1.0" 202 Accepted 2026-04-21 02:37:51,083 - lava_callback - INFO - Starting async job submit for node 69e43ff63551964617339626 2026-04-21 02:37:51,084 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:37:51,098 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43ff63551964617339626 HTTP/1.1" 200 None 2026-04-21 02:37:51,099 - lava_callback - DEBUG - Processing node 69e43ff63551964617339626: baseline-arm64-baylibre 2026-04-21 02:37:51,099 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e43ff63551964617339626 2026-04-21 02:37:51,100 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:37:51,113 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e43ff63551964617339626 HTTP/1.1" 200 None 2026-04-21 02:37:51,114 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:37:51,127 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 02:37:51,128 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:37:51,146 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e43ff63551964617339626 HTTP/1.1" 200 None 2026-04-21 02:37:51,147 - lava_callback - ERROR - Error processing callback for node 69e43ff63551964617339626: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:56134 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37336 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45498 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45512 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47030 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47036 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38190 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44920 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53444 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46162 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:39:11,164 - lava_callback - INFO - Received callback for node 69e440413551964617339645 INFO: 172.20.0.1:42502 - "POST /node/69e440413551964617339645 HTTP/1.0" 202 Accepted 2026-04-21 02:39:11,170 - lava_callback - INFO - Starting async job submit for node 69e440413551964617339645 2026-04-21 02:39:11,171 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:39:11,185 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e440413551964617339645 HTTP/1.1" 200 None 2026-04-21 02:39:11,186 - lava_callback - DEBUG - Processing node 69e440413551964617339645: baseline-arm64-baylibre 2026-04-21 02:39:11,186 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e440413551964617339645 2026-04-21 02:39:11,187 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:39:11,200 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e440413551964617339645 HTTP/1.1" 200 None 2026-04-21 02:39:11,202 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:39:11,215 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 02:39:11,216 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:39:11,236 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e440413551964617339645 HTTP/1.1" 200 None 2026-04-21 02:39:11,238 - lava_callback - ERROR - Error processing callback for node 69e440413551964617339645: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:42510 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:42514 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49148 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49152 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41580 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60384 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57556 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42500 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51080 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51090 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53442 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53456 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44650 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:40:37,427 - lava_callback - INFO - Received callback for node 69e44094355196461733966b INFO: 172.20.0.1:44658 - "POST /node/69e44094355196461733966b HTTP/1.0" 202 Accepted 2026-04-21 02:40:37,433 - lava_callback - INFO - Starting async job submit for node 69e44094355196461733966b 2026-04-21 02:40:37,434 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:40:37,447 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44094355196461733966b HTTP/1.1" 200 None 2026-04-21 02:40:37,448 - lava_callback - DEBUG - Processing node 69e44094355196461733966b: baseline-arm64-clabbe 2026-04-21 02:40:37,448 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44094355196461733966b 2026-04-21 02:40:37,449 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:40:37,462 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44094355196461733966b HTTP/1.1" 200 None 2026-04-21 02:40:37,463 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:40:37,479 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 02:40:37,480 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:40:37,498 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44094355196461733966b HTTP/1.1" 200 None 2026-04-21 02:40:37,499 - lava_callback - ERROR - Error processing callback for node 69e44094355196461733966b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:56024 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53008 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32896 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50854 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50856 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45938 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45952 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:41:31,249 - lava_callback - INFO - Received callback for node 69e440d7355196461733968e INFO: 172.20.0.1:46720 - "POST /node/69e440d7355196461733968e HTTP/1.0" 202 Accepted 2026-04-21 02:41:31,255 - lava_callback - INFO - Starting async job submit for node 69e440d7355196461733968e 2026-04-21 02:41:31,256 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:41:31,270 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e440d7355196461733968e HTTP/1.1" 200 None 2026-04-21 02:41:31,271 - lava_callback - DEBUG - Processing node 69e440d7355196461733968e: baseline-arm64-baylibre 2026-04-21 02:41:31,271 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e440d7355196461733968e 2026-04-21 02:41:31,271 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:41:31,285 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e440d7355196461733968e HTTP/1.1" 200 None 2026-04-21 02:41:31,286 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:41:31,300 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 02:41:31,301 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:41:31,318 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e440d7355196461733968e HTTP/1.1" 200 None 2026-04-21 02:41:31,319 - lava_callback - ERROR - Error processing callback for node 69e440d7355196461733968e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:46724 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59564 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53314 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:42:01,305 - lava_callback - INFO - Received callback for node 69e6e3e760f5826562c16928 INFO: 172.20.0.1:41344 - "POST /node/69e6e3e760f5826562c16928 HTTP/1.0" 202 Accepted 2026-04-21 02:42:01,311 - lava_callback - INFO - Starting async job submit for node 69e6e3e760f5826562c16928 2026-04-21 02:42:01,313 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:42:01,328 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6e3e760f5826562c16928 HTTP/1.1" 200 None 2026-04-21 02:42:01,329 - lava_callback - DEBUG - Processing node 69e6e3e760f5826562c16928: baseline-nfs-arm64 2026-04-21 02:42:01,330 - lava_callback - WARNING - Failed to upload log 2026-04-21 02:42:01,331 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:42:01,444 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:42:01,446 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6e3e760f5826562c16928/lava-logs.yaml 2026-04-21 02:42:01,447 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:42:01,712 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:42:01,715 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6e3e760f5826562c16928/lava_callback.json.gz 2026-04-21 02:42:01,718 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:42:01,731 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6e3e760f5826562c16928 HTTP/1.1" 200 None 2026-04-21 02:42:01,732 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:42:01,746 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 02:42:01,748 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:42:01,767 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6e3e760f5826562c16928 HTTP/1.1" 200 None 2026-04-21 02:42:01,770 - lava_callback - INFO - Completed processing callback for node 69e6e3e760f5826562c16928 INFO: 172.20.0.1:41352 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37016 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 02:42:15,617 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:42:15,631 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:37020 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60986 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60994 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37118 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44496 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55214 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:42:57,544 - lava_callback - INFO - Received callback for node 69e4412a35519646173396b8 INFO: 172.20.0.1:55220 - "POST /node/69e4412a35519646173396b8 HTTP/1.0" 202 Accepted 2026-04-21 02:42:57,550 - lava_callback - INFO - Starting async job submit for node 69e4412a35519646173396b8 2026-04-21 02:42:57,551 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:42:57,565 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4412a35519646173396b8 HTTP/1.1" 200 None 2026-04-21 02:42:57,565 - lava_callback - DEBUG - Processing node 69e4412a35519646173396b8: baseline-arm64-clabbe 2026-04-21 02:42:57,566 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4412a35519646173396b8 2026-04-21 02:42:57,566 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:42:57,593 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4412a35519646173396b8 HTTP/1.1" 200 None 2026-04-21 02:42:57,595 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:42:57,608 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 02:42:57,609 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:42:57,626 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4412a35519646173396b8 HTTP/1.1" 200 None 2026-04-21 02:42:57,627 - lava_callback - ERROR - Error processing callback for node 69e4412a35519646173396b8: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49740 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55132 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:55138 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33828 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33842 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35242 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60290 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54402 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43534 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:44:11,533 - lava_callback - INFO - Received callback for node 69e4416f35519646173396d9 INFO: 172.20.0.1:47616 - "POST /node/69e4416f35519646173396d9 HTTP/1.0" 202 Accepted 2026-04-21 02:44:11,543 - lava_callback - INFO - Starting async job submit for node 69e4416f35519646173396d9 2026-04-21 02:44:11,544 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:44:11,567 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4416f35519646173396d9 HTTP/1.1" 200 None 2026-04-21 02:44:11,568 - lava_callback - DEBUG - Processing node 69e4416f35519646173396d9: baseline-arm64-baylibre 2026-04-21 02:44:11,568 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4416f35519646173396d9 2026-04-21 02:44:11,569 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:44:11,583 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4416f35519646173396d9 HTTP/1.1" 200 None 2026-04-21 02:44:11,585 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:44:11,601 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 02:44:11,603 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:44:11,643 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4416f35519646173396d9 HTTP/1.1" 200 None 2026-04-21 02:44:11,645 - lava_callback - ERROR - Error processing callback for node 69e4416f35519646173396d9: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47620 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47622 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38948 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38964 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39506 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40616 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58058 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33032 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56856 - "GET /node/69b12a09a870a1dc3343bb60%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:56864 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:56868 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40502 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40506 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58704 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:45:37,614 - lava_callback - INFO - Received callback for node 69e441c03551964617339702 INFO: 172.20.0.1:58718 - "POST /node/69e441c03551964617339702 HTTP/1.0" 202 Accepted 2026-04-21 02:45:37,619 - lava_callback - INFO - Starting async job submit for node 69e441c03551964617339702 2026-04-21 02:45:37,620 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:45:37,635 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e441c03551964617339702 HTTP/1.1" 200 None 2026-04-21 02:45:37,636 - lava_callback - DEBUG - Processing node 69e441c03551964617339702: baseline-arm64-clabbe 2026-04-21 02:45:37,636 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e441c03551964617339702 2026-04-21 02:45:37,637 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:45:37,649 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e441c03551964617339702 HTTP/1.1" 200 None 2026-04-21 02:45:37,650 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:45:37,668 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 02:45:37,669 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:45:37,686 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e441c03551964617339702 HTTP/1.1" 200 None 2026-04-21 02:45:37,687 - lava_callback - ERROR - Error processing callback for node 69e441c03551964617339702: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:32770 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51578 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53632 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43064 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43070 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35170 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35176 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:46:33,026 - lava_callback - INFO - Received callback for node 69e6e4d760f5826562c169a6 INFO: 172.20.0.1:35364 - "POST /node/69e6e4d760f5826562c169a6 HTTP/1.0" 202 Accepted 2026-04-21 02:46:33,033 - lava_callback - INFO - Starting async job submit for node 69e6e4d760f5826562c169a6 2026-04-21 02:46:33,064 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:46:33,084 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6e4d760f5826562c169a6 HTTP/1.1" 200 None 2026-04-21 02:46:33,085 - lava_callback - DEBUG - Processing node 69e6e4d760f5826562c169a6: baseline-arm64 2026-04-21 02:46:33,225 - lava_callback - WARNING - Failed to upload log 2026-04-21 02:46:33,304 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:46:33,473 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:46:33,475 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6e4d760f5826562c169a6/lava-logs.yaml 2026-04-21 02:46:33,478 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:46:33,749 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:46:33,751 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6e4d760f5826562c169a6/lava_callback.json.gz 2026-04-21 02:46:33,781 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:46:33,800 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6e4d760f5826562c169a6 HTTP/1.1" 200 None 2026-04-21 02:46:33,802 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:46:33,815 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 02:46:33,817 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:46:33,834 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6e4d760f5826562c169a6 HTTP/1.1" 200 None 2026-04-21 02:46:33,886 - lava_callback - INFO - Completed processing callback for node 69e6e4d760f5826562c169a6 INFO: 172.20.0.1:35378 - "GET /node/6990229976d4b6342e9df5b1%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:35384 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:46:45,634 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:46:45,650 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:60608 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:46:51,586 - lava_callback - INFO - Received callback for node 69e442063551964617339724 INFO: 172.20.0.1:35528 - "POST /node/69e442063551964617339724 HTTP/1.0" 202 Accepted 2026-04-21 02:46:51,592 - lava_callback - INFO - Starting async job submit for node 69e442063551964617339724 2026-04-21 02:46:51,592 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:46:51,607 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e442063551964617339724 HTTP/1.1" 200 None 2026-04-21 02:46:51,608 - lava_callback - DEBUG - Processing node 69e442063551964617339724: baseline-arm64-baylibre 2026-04-21 02:46:51,608 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e442063551964617339724 2026-04-21 02:46:51,608 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:46:51,621 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e442063551964617339724 HTTP/1.1" 200 None 2026-04-21 02:46:51,622 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:46:51,635 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 02:46:51,636 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:46:51,653 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e442063551964617339724 HTTP/1.1" 200 None 2026-04-21 02:46:51,655 - lava_callback - ERROR - Error processing callback for node 69e442063551964617339724: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:35540 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59702 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49070 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49086 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49066 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49076 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45928 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52692 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36384 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:47:57,729 - lava_callback - INFO - Received callback for node 69e44256355196461733974c INFO: 172.20.0.1:36390 - "POST /node/69e44256355196461733974c HTTP/1.0" 202 Accepted 2026-04-21 02:47:57,735 - lava_callback - INFO - Starting async job submit for node 69e44256355196461733974c 2026-04-21 02:47:57,736 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:47:57,751 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44256355196461733974c HTTP/1.1" 200 None 2026-04-21 02:47:57,752 - lava_callback - DEBUG - Processing node 69e44256355196461733974c: baseline-arm64-clabbe 2026-04-21 02:47:57,752 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44256355196461733974c 2026-04-21 02:47:57,753 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:47:57,766 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44256355196461733974c HTTP/1.1" 200 None 2026-04-21 02:47:57,767 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:47:57,788 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 02:47:57,791 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:47:57,826 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44256355196461733974c HTTP/1.1" 200 None 2026-04-21 02:47:57,827 - lava_callback - ERROR - Error processing callback for node 69e44256355196461733974c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:33836 - "GET /node/69dbcb0b6a0b8baac20cf304%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:33844 - "GET /node/698ee482f0e3ffbba67d209b%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:33858 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53778 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53782 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43840 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43844 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58616 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:48:45,311 - lava_callback - INFO - Received callback for node 69e6e52860f5826562c169e4 INFO: 172.20.0.1:46648 - "POST /node/69e6e52860f5826562c169e4 HTTP/1.0" 202 Accepted 2026-04-21 02:48:45,318 - lava_callback - INFO - Starting async job submit for node 69e6e52860f5826562c169e4 2026-04-21 02:48:45,351 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:48:45,365 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6e52860f5826562c169e4 HTTP/1.1" 200 None 2026-04-21 02:48:45,366 - lava_callback - DEBUG - Processing node 69e6e52860f5826562c169e4: baseline-arm64 2026-04-21 02:48:45,585 - lava_callback - WARNING - Failed to upload log 2026-04-21 02:48:45,681 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:48:45,836 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:48:45,838 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6e52860f5826562c169e4/lava-logs.yaml 2026-04-21 02:48:45,842 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:48:45,955 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:48:45,957 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6e52860f5826562c169e4/lava_callback.json.gz 2026-04-21 02:48:45,987 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:48:46,001 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6e52860f5826562c169e4 HTTP/1.1" 200 None 2026-04-21 02:48:46,003 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:48:46,017 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 02:48:46,019 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:48:46,065 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6e52860f5826562c169e4 HTTP/1.1" 200 None 2026-04-21 02:48:46,143 - lava_callback - INFO - Completed processing callback for node 69e6e52860f5826562c169e4 INFO: 172.20.0.1:46654 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49384 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40274 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:49:11,828 - lava_callback - INFO - Received callback for node 69e4429b355196461733976b INFO: 172.20.0.1:41628 - "POST /node/69e4429b355196461733976b HTTP/1.0" 202 Accepted 2026-04-21 02:49:11,834 - lava_callback - INFO - Starting async job submit for node 69e4429b355196461733976b 2026-04-21 02:49:11,835 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:49:11,855 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4429b355196461733976b HTTP/1.1" 200 None 2026-04-21 02:49:11,856 - lava_callback - DEBUG - Processing node 69e4429b355196461733976b: baseline-arm64-baylibre 2026-04-21 02:49:11,856 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4429b355196461733976b 2026-04-21 02:49:11,857 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:49:11,870 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4429b355196461733976b HTTP/1.1" 200 None 2026-04-21 02:49:11,871 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:49:11,884 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 02:49:11,885 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:49:11,904 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4429b355196461733976b HTTP/1.1" 200 None 2026-04-21 02:49:11,905 - lava_callback - ERROR - Error processing callback for node 69e4429b355196461733976b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:41644 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 02:49:15,654 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:49:15,670 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:41646 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56468 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56472 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57084 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45936 - "GET /node/693a190778d7681e6b4907a4%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:45948 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37946 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53266 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36932 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:36944 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33820 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33822 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49544 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:50:37,897 - lava_callback - INFO - Received callback for node 69e442ec355196461733978d INFO: 172.20.0.1:49558 - "POST /node/69e442ec355196461733978d HTTP/1.0" 202 Accepted 2026-04-21 02:50:37,903 - lava_callback - INFO - Starting async job submit for node 69e442ec355196461733978d 2026-04-21 02:50:37,904 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:50:37,924 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e442ec355196461733978d HTTP/1.1" 200 None 2026-04-21 02:50:37,924 - lava_callback - DEBUG - Processing node 69e442ec355196461733978d: baseline-arm64-clabbe 2026-04-21 02:50:37,925 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e442ec355196461733978d 2026-04-21 02:50:37,925 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:50:37,938 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e442ec355196461733978d HTTP/1.1" 200 None 2026-04-21 02:50:37,939 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:50:37,951 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 02:50:37,953 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:50:37,972 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e442ec355196461733978d HTTP/1.1" 200 None 2026-04-21 02:50:37,974 - lava_callback - ERROR - Error processing callback for node 69e442ec355196461733978d: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:58028 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51204 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57226 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37564 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37568 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53988 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53998 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48976 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57806 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42824 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:51:57,915 - lava_callback - INFO - Received callback for node 69e4433735519646173397a8 INFO: 172.20.0.1:42836 - "POST /node/69e4433735519646173397a8 HTTP/1.0" 202 Accepted 2026-04-21 02:51:57,920 - lava_callback - INFO - Starting async job submit for node 69e4433735519646173397a8 2026-04-21 02:51:57,921 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:51:57,944 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4433735519646173397a8 HTTP/1.1" 200 None 2026-04-21 02:51:57,945 - lava_callback - DEBUG - Processing node 69e4433735519646173397a8: baseline-arm64-clabbe 2026-04-21 02:51:57,945 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4433735519646173397a8 2026-04-21 02:51:57,946 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:51:57,961 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4433735519646173397a8 HTTP/1.1" 200 None 2026-04-21 02:51:57,962 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:51:57,975 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 02:51:57,976 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:51:57,993 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4433735519646173397a8 HTTP/1.1" 200 None 2026-04-21 02:51:57,994 - lava_callback - ERROR - Error processing callback for node 69e4433735519646173397a8: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:37256 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53232 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53236 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50512 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50524 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52168 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43338 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55360 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44336 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54358 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:54368 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36728 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36730 - "GET /node/693f6f5e0d8ce4018da92d49%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:36744 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60322 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32884 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36682 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38256 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52468 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52478 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:54:17,955 - lava_callback - INFO - Received callback for node 69e443cb35519646173397dc INFO: 172.20.0.1:52490 - "POST /node/69e443cb35519646173397dc HTTP/1.0" 202 Accepted 2026-04-21 02:54:17,961 - lava_callback - INFO - Starting async job submit for node 69e443cb35519646173397dc 2026-04-21 02:54:17,962 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:54:17,976 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e443cb35519646173397dc HTTP/1.1" 200 None 2026-04-21 02:54:17,977 - lava_callback - DEBUG - Processing node 69e443cb35519646173397dc: baseline-arm64-clabbe 2026-04-21 02:54:17,977 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e443cb35519646173397dc 2026-04-21 02:54:17,977 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:54:17,990 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e443cb35519646173397dc HTTP/1.1" 200 None 2026-04-21 02:54:17,991 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:54:18,004 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 02:54:18,005 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:54:18,036 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e443cb35519646173397dc HTTP/1.1" 200 None 2026-04-21 02:54:18,037 - lava_callback - ERROR - Error processing callback for node 69e443cb35519646173397dc: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49864 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49872 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52956 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32796 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45098 - "GET /node/698ea435db401c547d571ae2%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:45104 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48024 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:55:12,261 - lava_callback - INFO - Received callback for node 69e4440635519646173397f8 INFO: 172.20.0.1:47982 - "POST /node/69e4440635519646173397f8 HTTP/1.0" 202 Accepted 2026-04-21 02:55:12,268 - lava_callback - INFO - Starting async job submit for node 69e4440635519646173397f8 2026-04-21 02:55:12,270 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:55:12,283 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4440635519646173397f8 HTTP/1.1" 200 None 2026-04-21 02:55:12,284 - lava_callback - DEBUG - Processing node 69e4440635519646173397f8: baseline-arm-baylibre 2026-04-21 02:55:12,284 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4440635519646173397f8 2026-04-21 02:55:12,285 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:55:12,297 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4440635519646173397f8 HTTP/1.1" 200 None 2026-04-21 02:55:12,299 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:55:12,313 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e33e283745d1193f22738a HTTP/1.1" 200 None 2026-04-21 02:55:12,314 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:55:12,340 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4440635519646173397f8 HTTP/1.1" 200 None 2026-04-21 02:55:12,341 - lava_callback - ERROR - Error processing callback for node 69e4440635519646173397f8: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47984 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47988 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49150 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49154 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42758 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58824 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:55:52,009 - lava_callback - INFO - Received callback for node 69e4442a3551964617339807 INFO: 172.20.0.1:43638 - "POST /node/69e4442a3551964617339807 HTTP/1.0" 202 Accepted 2026-04-21 02:55:52,015 - lava_callback - INFO - Starting async job submit for node 69e4442a3551964617339807 2026-04-21 02:55:52,016 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:55:52,030 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4442a3551964617339807 HTTP/1.1" 200 None 2026-04-21 02:55:52,031 - lava_callback - DEBUG - Processing node 69e4442a3551964617339807: baseline-arm-baylibre 2026-04-21 02:55:52,031 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4442a3551964617339807 2026-04-21 02:55:52,032 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:55:52,046 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4442a3551964617339807 HTTP/1.1" 200 None 2026-04-21 02:55:52,047 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:55:52,061 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e33e283745d1193f22738a HTTP/1.1" 200 None 2026-04-21 02:55:52,062 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:55:52,080 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4442a3551964617339807 HTTP/1.1" 200 None 2026-04-21 02:55:52,081 - lava_callback - ERROR - Error processing callback for node 69e4442a3551964617339807: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:43642 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40300 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42072 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:42088 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60028 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60044 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43940 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:56:38,079 - lava_callback - INFO - Received callback for node 69e444613551964617339810 INFO: 172.20.0.1:43956 - "POST /node/69e444613551964617339810 HTTP/1.0" 202 Accepted 2026-04-21 02:56:38,084 - lava_callback - INFO - Starting async job submit for node 69e444613551964617339810 2026-04-21 02:56:38,086 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:56:38,103 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e444613551964617339810 HTTP/1.1" 200 None 2026-04-21 02:56:38,104 - lava_callback - DEBUG - Processing node 69e444613551964617339810: baseline-arm64-clabbe 2026-04-21 02:56:38,104 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e444613551964617339810 2026-04-21 02:56:38,105 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:56:38,117 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e444613551964617339810 HTTP/1.1" 200 None 2026-04-21 02:56:38,119 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:56:38,131 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 02:56:38,133 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:56:38,153 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e444613551964617339810 HTTP/1.1" 200 None 2026-04-21 02:56:38,154 - lava_callback - ERROR - Error processing callback for node 69e444613551964617339810: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:34306 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34090 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46800 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41770 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41772 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42548 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42558 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37266 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51888 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:57:52,173 - lava_callback - INFO - Received callback for node 69e4449b355196461733982c INFO: 172.20.0.1:55832 - "POST /node/69e4449b355196461733982c HTTP/1.0" 202 Accepted 2026-04-21 02:57:52,178 - lava_callback - INFO - Starting async job submit for node 69e4449b355196461733982c 2026-04-21 02:57:52,179 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:57:52,194 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4449b355196461733982c HTTP/1.1" 200 None 2026-04-21 02:57:52,195 - lava_callback - DEBUG - Processing node 69e4449b355196461733982c: baseline-arm-baylibre 2026-04-21 02:57:52,195 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4449b355196461733982c 2026-04-21 02:57:52,196 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:57:52,208 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4449b355196461733982c HTTP/1.1" 200 None 2026-04-21 02:57:52,210 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:57:52,223 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e33e283745d1193f22738a HTTP/1.1" 200 None 2026-04-21 02:57:52,225 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:57:52,243 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4449b355196461733982c HTTP/1.1" 200 None 2026-04-21 02:57:52,244 - lava_callback - ERROR - Error processing callback for node 69e4449b355196461733982c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:55848 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53182 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52392 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52408 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48598 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48600 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41850 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:58:38,150 - lava_callback - INFO - Received callback for node 69e444cf355196461733983b INFO: 172.20.0.1:41854 - "POST /node/69e444cf355196461733983b HTTP/1.0" 202 Accepted 2026-04-21 02:58:38,155 - lava_callback - INFO - Starting async job submit for node 69e444cf355196461733983b 2026-04-21 02:58:38,156 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:58:38,171 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e444cf355196461733983b HTTP/1.1" 200 None 2026-04-21 02:58:38,171 - lava_callback - DEBUG - Processing node 69e444cf355196461733983b: baseline-arm64-clabbe 2026-04-21 02:58:38,172 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e444cf355196461733983b 2026-04-21 02:58:38,172 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:58:38,185 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e444cf355196461733983b HTTP/1.1" 200 None 2026-04-21 02:58:38,186 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:58:38,199 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 02:58:38,200 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:58:38,217 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e444cf355196461733983b HTTP/1.1" 200 None 2026-04-21 02:58:38,218 - lava_callback - ERROR - Error processing callback for node 69e444cf355196461733983b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:52646 - "GET / HTTP/1.0" 200 OK 2026-04-21 02:58:49,999 - lava_callback - INFO - Received callback for node 69e6e57b60f5826562c16a18 INFO: 172.20.0.1:52464 - "POST /node/69e6e57b60f5826562c16a18 HTTP/1.0" 202 Accepted 2026-04-21 02:58:50,006 - lava_callback - INFO - Starting async job submit for node 69e6e57b60f5826562c16a18 2026-04-21 02:58:50,031 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:58:50,047 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6e57b60f5826562c16a18 HTTP/1.1" 200 None 2026-04-21 02:58:50,048 - lava_callback - DEBUG - Processing node 69e6e57b60f5826562c16a18: baseline-arm64 2026-04-21 02:58:50,601 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:58:50,716 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:58:50,717 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6e57b60f5826562c16a18/log.txt.gz 2026-04-21 02:58:50,966 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:58:51,156 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:58:51,158 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6e57b60f5826562c16a18/lava-logs.yaml 2026-04-21 02:58:51,170 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 02:58:51,452 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 02:58:51,454 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6e57b60f5826562c16a18/lava_callback.json.gz 2026-04-21 02:58:51,479 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:58:51,496 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6e57b60f5826562c16a18 HTTP/1.1" 200 None 2026-04-21 02:58:51,498 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:58:51,512 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 02:58:51,513 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:58:51,531 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6e57b60f5826562c16a18 HTTP/1.1" 200 None 2026-04-21 02:58:51,590 - lava_callback - INFO - Completed processing callback for node 69e6e57b60f5826562c16a18 INFO: 172.20.0.1:52476 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56344 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58776 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 02:59:15,674 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 02:59:15,698 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:58780 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48556 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48568 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46802 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36848 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50656 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37930 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56690 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:56696 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38330 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38336 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48334 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47078 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57554 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:00:58,209 - lava_callback - INFO - Received callback for node 69e445633551964617339870 INFO: 172.20.0.1:57566 - "POST /node/69e445633551964617339870 HTTP/1.0" 202 Accepted 2026-04-21 03:00:58,215 - lava_callback - INFO - Starting async job submit for node 69e445633551964617339870 2026-04-21 03:00:58,216 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:00:58,239 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e445633551964617339870 HTTP/1.1" 200 None 2026-04-21 03:00:58,240 - lava_callback - DEBUG - Processing node 69e445633551964617339870: baseline-arm64-clabbe 2026-04-21 03:00:58,240 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e445633551964617339870 2026-04-21 03:00:58,241 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:00:58,273 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e445633551964617339870 HTTP/1.1" 200 None 2026-04-21 03:00:58,275 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:00:58,294 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 03:00:58,295 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:00:58,313 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e445633551964617339870 HTTP/1.1" 200 None 2026-04-21 03:00:58,315 - lava_callback - ERROR - Error processing callback for node 69e445633551964617339870: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:51930 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60624 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60628 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45386 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45394 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51786 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42702 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35436 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:01:58,256 - lava_callback - INFO - Received callback for node 69e445a1355196461733988b INFO: 172.20.0.1:35440 - "POST /node/69e445a1355196461733988b HTTP/1.0" 202 Accepted 2026-04-21 03:01:58,262 - lava_callback - INFO - Starting async job submit for node 69e445a1355196461733988b 2026-04-21 03:01:58,263 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:01:58,277 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e445a1355196461733988b HTTP/1.1" 200 None 2026-04-21 03:01:58,278 - lava_callback - DEBUG - Processing node 69e445a1355196461733988b: baseline-arm-clabbe 2026-04-21 03:01:58,278 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e445a1355196461733988b 2026-04-21 03:01:58,279 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:01:58,296 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e445a1355196461733988b HTTP/1.1" 200 None 2026-04-21 03:01:58,297 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:01:58,310 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e33e283745d1193f22738a HTTP/1.1" 200 None 2026-04-21 03:01:58,311 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:01:58,329 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e445a1355196461733988b HTTP/1.1" 200 None 2026-04-21 03:01:58,329 - lava_callback - ERROR - Error processing callback for node 69e445a1355196461733988b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:56314 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41222 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41224 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37934 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37944 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40522 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:02:38,243 - lava_callback - INFO - Received callback for node 69e445c53551964617339896 INFO: 172.20.0.1:40538 - "POST /node/69e445c53551964617339896 HTTP/1.0" 202 Accepted 2026-04-21 03:02:38,249 - lava_callback - INFO - Starting async job submit for node 69e445c53551964617339896 2026-04-21 03:02:38,250 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:02:38,263 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e445c53551964617339896 HTTP/1.1" 200 None 2026-04-21 03:02:38,264 - lava_callback - DEBUG - Processing node 69e445c53551964617339896: baseline-arm-clabbe 2026-04-21 03:02:38,264 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e445c53551964617339896 2026-04-21 03:02:38,265 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:02:38,278 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e445c53551964617339896 HTTP/1.1" 200 None 2026-04-21 03:02:38,280 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:02:38,293 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e33e283745d1193f22738a HTTP/1.1" 200 None 2026-04-21 03:02:38,294 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:02:38,312 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e445c53551964617339896 HTTP/1.1" 200 None 2026-04-21 03:02:38,313 - lava_callback - ERROR - Error processing callback for node 69e445c53551964617339896: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:38108 - "GET /node/698b8f210a16aea5a0dc8ecd%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:38110 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43408 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45598 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57790 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57794 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:03:18,280 - lava_callback - INFO - Received callback for node 69e445ea35519646173398a1 INFO: 172.20.0.1:57806 - "POST /node/69e445ea35519646173398a1 HTTP/1.0" 202 Accepted 2026-04-21 03:03:18,286 - lava_callback - INFO - Starting async job submit for node 69e445ea35519646173398a1 2026-04-21 03:03:18,287 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:03:18,301 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e445ea35519646173398a1 HTTP/1.1" 200 None 2026-04-21 03:03:18,302 - lava_callback - DEBUG - Processing node 69e445ea35519646173398a1: baseline-arm-clabbe 2026-04-21 03:03:18,302 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e445ea35519646173398a1 2026-04-21 03:03:18,303 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:03:18,326 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e445ea35519646173398a1 HTTP/1.1" 200 None 2026-04-21 03:03:18,328 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:03:18,342 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e33e283745d1193f22738a HTTP/1.1" 200 None 2026-04-21 03:03:18,343 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:03:18,368 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e445ea35519646173398a1 HTTP/1.1" 200 None 2026-04-21 03:03:18,369 - lava_callback - ERROR - Error processing callback for node 69e445ea35519646173398a1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:37516 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37518 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:03:32,734 - lava_callback - INFO - Received callback for node 69e6e73d60f5826562c16b3b INFO: 172.20.0.1:45776 - "POST /node/69e6e73d60f5826562c16b3b HTTP/1.0" 202 Accepted 2026-04-21 03:03:32,741 - lava_callback - INFO - Starting async job submit for node 69e6e73d60f5826562c16b3b 2026-04-21 03:03:32,779 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:03:32,795 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6e73d60f5826562c16b3b HTTP/1.1" 200 None 2026-04-21 03:03:32,796 - lava_callback - DEBUG - Processing node 69e6e73d60f5826562c16b3b: baseline-arm64 2026-04-21 03:03:33,165 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:03:33,445 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:03:33,447 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6e73d60f5826562c16b3b/log.txt.gz 2026-04-21 03:03:33,665 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:03:34,013 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:03:34,015 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6e73d60f5826562c16b3b/lava-logs.yaml 2026-04-21 03:03:34,024 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:03:34,125 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:03:34,127 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6e73d60f5826562c16b3b/lava_callback.json.gz DEBUG: Setup failed for 69e6e73d60f5826562c16b3b. Transitting job node result: incomplete -> fail 2026-04-21 03:03:34,198 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:03:34,212 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6e73d60f5826562c16b3b HTTP/1.1" 200 None 2026-04-21 03:03:34,214 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:03:34,226 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 03:03:34,228 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:03:34,247 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6e73d60f5826562c16b3b HTTP/1.1" 200 None 2026-04-21 03:03:34,324 - lava_callback - INFO - Completed processing callback for node 69e6e73d60f5826562c16b3b INFO: 172.20.0.1:34710 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:03:45,701 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:03:45,719 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:34718 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56100 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51360 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38036 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:38038 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:04:18,356 - lava_callback - INFO - Received callback for node 69e4461c35519646173398af INFO: 172.20.0.1:38054 - "POST /node/69e4461c35519646173398af HTTP/1.0" 202 Accepted 2026-04-21 03:04:18,362 - lava_callback - INFO - Starting async job submit for node 69e4461c35519646173398af 2026-04-21 03:04:18,363 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:04:18,379 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4461c35519646173398af HTTP/1.1" 200 None 2026-04-21 03:04:18,380 - lava_callback - DEBUG - Processing node 69e4461c35519646173398af: baseline-arm64-clabbe 2026-04-21 03:04:18,380 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4461c35519646173398af 2026-04-21 03:04:18,381 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:04:18,394 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4461c35519646173398af HTTP/1.1" 200 None 2026-04-21 03:04:18,396 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:04:18,409 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 03:04:18,410 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:04:18,428 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4461c35519646173398af HTTP/1.1" 200 None 2026-04-21 03:04:18,429 - lava_callback - ERROR - Error processing callback for node 69e4461c35519646173398af: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:38234 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38240 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38094 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52738 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44068 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49426 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43866 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43882 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:05:18,385 - lava_callback - INFO - Received callback for node 69e4465a35519646173398cc INFO: 172.20.0.1:43898 - "POST /node/69e4465a35519646173398cc HTTP/1.0" 202 Accepted 2026-04-21 03:05:18,394 - lava_callback - INFO - Starting async job submit for node 69e4465a35519646173398cc 2026-04-21 03:05:18,395 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:05:18,411 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4465a35519646173398cc HTTP/1.1" 200 None 2026-04-21 03:05:18,412 - lava_callback - DEBUG - Processing node 69e4465a35519646173398cc: baseline-arm-clabbe 2026-04-21 03:05:18,412 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4465a35519646173398cc 2026-04-21 03:05:18,413 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:05:18,429 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4465a35519646173398cc HTTP/1.1" 200 None 2026-04-21 03:05:18,430 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:05:18,443 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e33e283745d1193f22738a HTTP/1.1" 200 None 2026-04-21 03:05:18,445 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:05:18,462 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4465a35519646173398cc HTTP/1.1" 200 None 2026-04-21 03:05:18,463 - lava_callback - ERROR - Error processing callback for node 69e4465a35519646173398cc: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:55380 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55396 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47948 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:05:38,332 - lava_callback - INFO - Received callback for node 69e4467e35519646173398d8 INFO: 172.20.0.1:47952 - "POST /node/69e4467e35519646173398d8 HTTP/1.0" 202 Accepted 2026-04-21 03:05:38,338 - lava_callback - INFO - Starting async job submit for node 69e4467e35519646173398d8 2026-04-21 03:05:38,339 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:05:38,353 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4467e35519646173398d8 HTTP/1.1" 200 None 2026-04-21 03:05:38,354 - lava_callback - DEBUG - Processing node 69e4467e35519646173398d8: baseline-arm-clabbe 2026-04-21 03:05:38,354 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4467e35519646173398d8 2026-04-21 03:05:38,354 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:05:38,368 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4467e35519646173398d8 HTTP/1.1" 200 None 2026-04-21 03:05:38,370 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:05:38,383 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e33e283745d1193f22738a HTTP/1.1" 200 None 2026-04-21 03:05:38,385 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:05:38,408 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4467e35519646173398d8 HTTP/1.1" 200 None 2026-04-21 03:05:38,409 - lava_callback - ERROR - Error processing callback for node 69e4467e35519646173398d8: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:58488 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60206 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48122 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52274 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52278 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:06:18,366 - lava_callback - INFO - Received callback for node 69e446a335519646173398e5 INFO: 172.20.0.1:52288 - "POST /node/69e446a335519646173398e5 HTTP/1.0" 202 Accepted 2026-04-21 03:06:18,372 - lava_callback - INFO - Starting async job submit for node 69e446a335519646173398e5 2026-04-21 03:06:18,373 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:06:18,387 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e446a335519646173398e5 HTTP/1.1" 200 None 2026-04-21 03:06:18,388 - lava_callback - DEBUG - Processing node 69e446a335519646173398e5: baseline-arm-clabbe 2026-04-21 03:06:18,388 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e446a335519646173398e5 2026-04-21 03:06:18,389 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:06:18,401 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e446a335519646173398e5 HTTP/1.1" 200 None 2026-04-21 03:06:18,403 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:06:18,416 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e33e283745d1193f22738a HTTP/1.1" 200 None 2026-04-21 03:06:18,418 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:06:18,434 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e446a335519646173398e5 HTTP/1.1" 200 None 2026-04-21 03:06:18,436 - lava_callback - ERROR - Error processing callback for node 69e446a335519646173398e5: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:35684 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35700 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34178 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:06:44,801 - lava_callback - INFO - Received callback for node 69e678c0dc041e577b77b2f6 INFO: 172.20.0.1:46946 - "POST /node/69e678c0dc041e577b77b2f6 HTTP/1.0" 202 Accepted 2026-04-21 03:06:44,808 - lava_callback - INFO - Starting async job submit for node 69e678c0dc041e577b77b2f6 2026-04-21 03:06:44,830 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:06:44,845 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e678c0dc041e577b77b2f6 HTTP/1.1" 200 None 2026-04-21 03:06:44,846 - lava_callback - DEBUG - Processing node 69e678c0dc041e577b77b2f6: baseline-arm64-broonie 2026-04-21 03:06:45,128 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:06:45,241 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:06:45,243 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e678c0dc041e577b77b2f6/log.txt.gz 2026-04-21 03:06:45,410 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:06:45,587 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:06:45,589 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e678c0dc041e577b77b2f6/lava-logs.yaml 2026-04-21 03:06:45,596 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:06:45,711 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:06:45,713 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e678c0dc041e577b77b2f6/lava_callback.json.gz 2026-04-21 03:06:45,755 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:06:45,777 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e678c0dc041e577b77b2f6 HTTP/1.1" 200 None 2026-04-21 03:06:45,779 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:06:45,799 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e66a77dc041e577b7785d5 HTTP/1.1" 200 None 2026-04-21 03:06:45,801 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:06:45,823 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e678c0dc041e577b77b2f6 HTTP/1.1" 200 None 2026-04-21 03:06:45,824 - lava_callback - INFO - Completed processing callback for node 69e678c0dc041e577b77b2f6 INFO: 172.20.0.1:46956 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57314 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:06:58,388 - lava_callback - INFO - Received callback for node 69e446c835519646173398ee INFO: 172.20.0.1:57320 - "POST /node/69e446c835519646173398ee HTTP/1.0" 202 Accepted 2026-04-21 03:06:58,397 - lava_callback - INFO - Starting async job submit for node 69e446c835519646173398ee 2026-04-21 03:06:58,399 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:06:58,415 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e446c835519646173398ee HTTP/1.1" 200 None 2026-04-21 03:06:58,416 - lava_callback - DEBUG - Processing node 69e446c835519646173398ee: baseline-arm-clabbe 2026-04-21 03:06:58,416 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e446c835519646173398ee 2026-04-21 03:06:58,417 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:06:58,434 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e446c835519646173398ee HTTP/1.1" 200 None 2026-04-21 03:06:58,435 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:06:58,456 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e33e283745d1193f22738a HTTP/1.1" 200 None 2026-04-21 03:06:58,457 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:06:58,475 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e446c835519646173398ee HTTP/1.1" 200 None 2026-04-21 03:06:58,477 - lava_callback - ERROR - Error processing callback for node 69e446c835519646173398ee: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:48590 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58096 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 03:07:15,726 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:07:15,742 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:58102 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55728 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55742 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48812 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:07:38,430 - lava_callback - INFO - Received callback for node 69e446ec35519646173398f9 INFO: 172.20.0.1:48828 - "POST /node/69e446ec35519646173398f9 HTTP/1.0" 202 Accepted 2026-04-21 03:07:38,435 - lava_callback - INFO - Starting async job submit for node 69e446ec35519646173398f9 2026-04-21 03:07:38,436 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:07:38,451 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e446ec35519646173398f9 HTTP/1.1" 200 None 2026-04-21 03:07:38,451 - lava_callback - DEBUG - Processing node 69e446ec35519646173398f9: baseline-arm-clabbe 2026-04-21 03:07:38,452 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e446ec35519646173398f9 2026-04-21 03:07:38,452 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:07:38,467 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e446ec35519646173398f9 HTTP/1.1" 200 None 2026-04-21 03:07:38,468 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:07:38,481 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e33e283745d1193f22738a HTTP/1.1" 200 None 2026-04-21 03:07:38,483 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:07:38,501 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e446ec35519646173398f9 HTTP/1.1" 200 None 2026-04-21 03:07:38,502 - lava_callback - ERROR - Error processing callback for node 69e446ec35519646173398f9: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:45786 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60872 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38732 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33302 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33314 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33320 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33330 - "GET /favicon.ico HTTP/1.0" 404 Not Found 2026-04-21 03:08:18,402 - lava_callback - INFO - Received callback for node 69e447103551964617339904 INFO: 172.20.0.1:33344 - "POST /node/69e447103551964617339904 HTTP/1.0" 202 Accepted 2026-04-21 03:08:18,408 - lava_callback - INFO - Starting async job submit for node 69e447103551964617339904 2026-04-21 03:08:18,409 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:08:18,423 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e447103551964617339904 HTTP/1.1" 200 None 2026-04-21 03:08:18,423 - lava_callback - DEBUG - Processing node 69e447103551964617339904: baseline-arm-clabbe 2026-04-21 03:08:18,424 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e447103551964617339904 2026-04-21 03:08:18,424 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:08:18,436 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e447103551964617339904 HTTP/1.1" 200 None 2026-04-21 03:08:18,438 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:08:18,450 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e33e283745d1193f22738a HTTP/1.1" 200 None 2026-04-21 03:08:18,452 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:08:18,469 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e447103551964617339904 HTTP/1.1" 200 None 2026-04-21 03:08:18,470 - lava_callback - ERROR - Error processing callback for node 69e447103551964617339904: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:35314 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35328 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36254 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:08:44,891 - lava_callback - INFO - Received callback for node 69e69242dc041e577b77c116 INFO: 172.20.0.1:53558 - "POST /node/69e69242dc041e577b77c116 HTTP/1.0" 202 Accepted 2026-04-21 03:08:44,897 - lava_callback - INFO - Starting async job submit for node 69e69242dc041e577b77c116 2026-04-21 03:08:44,920 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:08:44,934 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e69242dc041e577b77c116 HTTP/1.1" 200 None 2026-04-21 03:08:44,935 - lava_callback - DEBUG - Processing node 69e69242dc041e577b77c116: baseline-arm64-broonie 2026-04-21 03:08:45,221 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:08:45,492 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:08:45,494 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e69242dc041e577b77c116/log.txt.gz 2026-04-21 03:08:45,686 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:08:45,869 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:08:45,872 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e69242dc041e577b77c116/lava-logs.yaml 2026-04-21 03:08:45,879 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:08:45,982 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:08:45,983 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e69242dc041e577b77c116/lava_callback.json.gz 2026-04-21 03:08:46,006 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:08:46,019 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e69242dc041e577b77c116 HTTP/1.1" 200 None 2026-04-21 03:08:46,020 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:08:46,034 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6522047863bb428fe4af5 HTTP/1.1" 200 None 2026-04-21 03:08:46,035 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:08:46,061 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e69242dc041e577b77c116 HTTP/1.1" 200 None 2026-04-21 03:08:46,063 - lava_callback - INFO - Completed processing callback for node 69e69242dc041e577b77c116 INFO: 172.20.0.1:53562 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48932 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57210 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42364 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 03:09:15,744 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:09:15,759 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:42370 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:09:18,946 - lava_callback - INFO - Received callback for node 69e6e7cf60f5826562c16b68 INFO: 172.20.0.1:42376 - "POST /node/69e6e7cf60f5826562c16b68 HTTP/1.0" 202 Accepted 2026-04-21 03:09:18,953 - lava_callback - INFO - Starting async job submit for node 69e6e7cf60f5826562c16b68 2026-04-21 03:09:18,979 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:09:18,993 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6e7cf60f5826562c16b68 HTTP/1.1" 200 None 2026-04-21 03:09:18,994 - lava_callback - DEBUG - Processing node 69e6e7cf60f5826562c16b68: baseline-nfs-arm64 2026-04-21 03:09:19,608 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:09:19,857 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:09:19,859 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6e7cf60f5826562c16b68/log.txt.gz 2026-04-21 03:09:20,132 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:09:20,270 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:09:20,272 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6e7cf60f5826562c16b68/lava-logs.yaml 2026-04-21 03:09:20,284 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:09:20,426 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:09:20,428 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6e7cf60f5826562c16b68/lava_callback.json.gz 2026-04-21 03:09:20,454 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:09:20,468 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6e7cf60f5826562c16b68 HTTP/1.1" 200 None 2026-04-21 03:09:20,470 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:09:20,482 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 03:09:20,484 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:09:20,502 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6e7cf60f5826562c16b68 HTTP/1.1" 200 None 2026-04-21 03:09:20,569 - lava_callback - INFO - Completed processing callback for node 69e6e7cf60f5826562c16b68 INFO: 172.20.0.1:34286 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34290 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40974 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:09:45,761 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:09:45,777 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:42016 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56644 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:09:58,318 - lava_callback - INFO - Received callback for node 69e6e5cf60f5826562c16a46 INFO: 172.20.0.1:56658 - "POST /node/69e6e5cf60f5826562c16a46 HTTP/1.0" 202 Accepted 2026-04-21 03:09:58,326 - lava_callback - INFO - Starting async job submit for node 69e6e5cf60f5826562c16a46 2026-04-21 03:09:58,370 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:09:58,392 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6e5cf60f5826562c16a46 HTTP/1.1" 200 None 2026-04-21 03:09:58,393 - lava_callback - DEBUG - Processing node 69e6e5cf60f5826562c16a46: baseline-nfs-arm64 2026-04-21 03:09:59,543 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:09:59,682 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:09:59,684 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6e5cf60f5826562c16a46/log.txt.gz 2026-04-21 03:10:00,397 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:10:00,605 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:10:00,608 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6e5cf60f5826562c16a46/lava-logs.yaml 2026-04-21 03:10:00,636 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:10:00,821 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:10:00,823 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6e5cf60f5826562c16a46/lava_callback.json.gz DEBUG: Setup failed for 69e6e5cf60f5826562c16a46. Transitting job node result: incomplete -> fail 2026-04-21 03:10:00,923 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:10:00,948 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6e5cf60f5826562c16a46 HTTP/1.1" 200 None 2026-04-21 03:10:00,950 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:10:00,964 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 03:10:00,966 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:10:00,988 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6e5cf60f5826562c16a46 HTTP/1.1" 200 None 2026-04-21 03:10:01,082 - lava_callback - INFO - Completed processing callback for node 69e6e5cf60f5826562c16a46 INFO: 172.20.0.1:37254 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37772 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 03:10:15,779 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:10:15,796 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:37778 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33290 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33302 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:10:30,251 - lava_callback - INFO - Received callback for node 69e6e6bf60f5826562c16ad6 INFO: 172.20.0.1:38510 - "POST /node/69e6e6bf60f5826562c16ad6 HTTP/1.0" 202 Accepted 2026-04-21 03:10:30,259 - lava_callback - INFO - Starting async job submit for node 69e6e6bf60f5826562c16ad6 2026-04-21 03:10:30,320 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:10:30,334 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6e6bf60f5826562c16ad6 HTTP/1.1" 200 None 2026-04-21 03:10:30,335 - lava_callback - DEBUG - Processing node 69e6e6bf60f5826562c16ad6: baseline-arm 2026-04-21 03:10:30,927 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:10:31,011 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:10:31,012 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69e6e6bf60f5826562c16ad6/log.txt.gz 2026-04-21 03:10:31,370 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:10:31,547 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:10:31,550 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69e6e6bf60f5826562c16ad6/lava-logs.yaml 2026-04-21 03:10:31,565 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:10:31,699 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:10:31,700 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69e6e6bf60f5826562c16ad6/lava_callback.json.gz DEBUG: Setup failed for 69e6e6bf60f5826562c16ad6. Transitting job node result: incomplete -> fail 2026-04-21 03:10:31,813 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:10:31,827 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6e6bf60f5826562c16ad6 HTTP/1.1" 200 None 2026-04-21 03:10:31,829 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:10:31,842 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c20d60f5826562c1508b HTTP/1.1" 200 None 2026-04-21 03:10:31,843 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:10:31,869 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6e6bf60f5826562c16ad6 HTTP/1.1" 200 None 2026-04-21 03:10:31,962 - lava_callback - INFO - Completed processing callback for node 69e6e6bf60f5826562c16ad6 INFO: 172.20.0.1:38518 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:10:44,710 - lava_callback - INFO - Received callback for node 69e6cb9b60f5826562c15c6d INFO: 172.20.0.1:35994 - "POST /node/69e6cb9b60f5826562c15c6d HTTP/1.0" 202 Accepted 2026-04-21 03:10:44,716 - lava_callback - INFO - Starting async job submit for node 69e6cb9b60f5826562c15c6d 2026-04-21 03:10:44,740 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:10:44,753 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb9b60f5826562c15c6d HTTP/1.1" 200 None 2026-04-21 03:10:44,754 - lava_callback - DEBUG - Processing node 69e6cb9b60f5826562c15c6d: baseline-arm64-broonie 2026-04-21 03:10:45,036 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:10:45,147 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:10:45,149 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cb9b60f5826562c15c6d/log.txt.gz 2026-04-21 03:10:45,322 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:10:45,466 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:10:45,468 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cb9b60f5826562c15c6d/lava-logs.yaml 2026-04-21 03:10:45,475 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:10:45,626 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:10:45,628 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cb9b60f5826562c15c6d/lava_callback.json.gz 2026-04-21 03:10:45,651 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:10:45,665 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb9b60f5826562c15c6d HTTP/1.1" 200 None 2026-04-21 03:10:45,667 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:10:45,684 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 03:10:45,686 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:10:45,715 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb9b60f5826562c15c6d HTTP/1.1" 200 None 2026-04-21 03:10:45,717 - lava_callback - INFO - Completed processing callback for node 69e6cb9b60f5826562c15c6d 2026-04-21 03:10:45,798 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:10:45,819 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:36008 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51642 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50404 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54262 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:54270 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44254 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:11:25,292 - lava_callback - INFO - Received callback for node 69e6e62160f5826562c16a6f INFO: 172.20.0.1:44266 - "POST /node/69e6e62160f5826562c16a6f HTTP/1.0" 202 Accepted 2026-04-21 03:11:25,301 - lava_callback - INFO - Starting async job submit for node 69e6e62160f5826562c16a6f 2026-04-21 03:11:25,363 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:11:25,402 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6e62160f5826562c16a6f HTTP/1.1" 200 None 2026-04-21 03:11:25,404 - lava_callback - DEBUG - Processing node 69e6e62160f5826562c16a6f: baseline-nfs-arm64 INFO: 172.20.0.1:44280 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:11:26,571 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:11:26,730 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:11:26,732 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6e62160f5826562c16a6f/log.txt.gz 2026-04-21 03:11:27,411 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:11:27,784 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:11:27,787 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6e62160f5826562c16a6f/lava-logs.yaml 2026-04-21 03:11:27,809 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:11:27,964 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:11:27,966 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6e62160f5826562c16a6f/lava_callback.json.gz DEBUG: Setup failed for 69e6e62160f5826562c16a6f. Transitting job node result: incomplete -> fail 2026-04-21 03:11:28,053 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:11:28,067 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6e62160f5826562c16a6f HTTP/1.1" 200 None 2026-04-21 03:11:28,069 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:11:28,086 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 03:11:28,088 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:11:28,117 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6e62160f5826562c16a6f HTTP/1.1" 200 None 2026-04-21 03:11:28,224 - lava_callback - INFO - Completed processing callback for node 69e6e62160f5826562c16a6f INFO: 172.20.0.1:42138 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:11:45,822 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:11:45,840 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:38884 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42044 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33370 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:12:12,971 - lava_callback - INFO - Received callback for node 69e447ff355196461733991c INFO: 172.20.0.1:35130 - "POST /node/69e447ff355196461733991c HTTP/1.0" 202 Accepted 2026-04-21 03:12:12,976 - lava_callback - INFO - Starting async job submit for node 69e447ff355196461733991c 2026-04-21 03:12:12,977 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:12:12,991 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e447ff355196461733991c HTTP/1.1" 200 None 2026-04-21 03:12:12,992 - lava_callback - DEBUG - Processing node 69e447ff355196461733991c: baseline-arm64-baylibre 2026-04-21 03:12:12,992 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e447ff355196461733991c 2026-04-21 03:12:12,993 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:12:13,005 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e447ff355196461733991c HTTP/1.1" 200 None 2026-04-21 03:12:13,007 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:12:13,032 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 03:12:13,034 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:12:13,052 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e447ff355196461733991c HTTP/1.1" 200 None 2026-04-21 03:12:13,053 - lava_callback - ERROR - Error processing callback for node 69e447ff355196461733991c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:35144 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35156 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44488 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44496 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59754 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:12:44,768 - lava_callback - INFO - Received callback for node 69e6d57660f5826562c162a6 INFO: 172.20.0.1:50786 - "POST /node/69e6d57660f5826562c162a6 HTTP/1.0" 202 Accepted 2026-04-21 03:12:44,774 - lava_callback - INFO - Starting async job submit for node 69e6d57660f5826562c162a6 2026-04-21 03:12:44,797 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:12:44,811 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6d57660f5826562c162a6 HTTP/1.1" 200 None 2026-04-21 03:12:44,812 - lava_callback - DEBUG - Processing node 69e6d57660f5826562c162a6: baseline-arm64-broonie 2026-04-21 03:12:45,104 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:12:45,364 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:12:45,365 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6d57660f5826562c162a6/log.txt.gz 2026-04-21 03:12:45,536 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:12:45,869 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:12:45,871 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6d57660f5826562c162a6/lava-logs.yaml 2026-04-21 03:12:45,879 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:12:45,976 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:12:45,978 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6d57660f5826562c162a6/lava_callback.json.gz 2026-04-21 03:12:46,001 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:12:46,019 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6d57660f5826562c162a6 HTTP/1.1" 200 None 2026-04-21 03:12:46,021 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:12:46,034 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e66a77dc041e577b7785d5 HTTP/1.1" 200 None 2026-04-21 03:12:46,036 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:12:46,057 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6d57660f5826562c162a6 HTTP/1.1" 200 None 2026-04-21 03:12:46,059 - lava_callback - INFO - Completed processing callback for node 69e6d57660f5826562c162a6 INFO: 172.20.0.1:50792 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40564 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37478 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:13:08,346 - lava_callback - INFO - Received callback for node 69e6e73560f5826562c16b36 INFO: 172.20.0.1:37480 - "POST /node/69e6e73560f5826562c16b36 HTTP/1.0" 202 Accepted 2026-04-21 03:13:08,353 - lava_callback - INFO - Starting async job submit for node 69e6e73560f5826562c16b36 2026-04-21 03:13:08,417 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:13:08,432 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6e73560f5826562c16b36 HTTP/1.1" 200 None 2026-04-21 03:13:08,433 - lava_callback - DEBUG - Processing node 69e6e73560f5826562c16b36: ltp-smoke 2026-04-21 03:13:08,937 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:13:09,038 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:13:09,040 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-smoke-69e6e73560f5826562c16b36/log.txt.gz 2026-04-21 03:13:09,305 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:13:09,462 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:13:09,464 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-smoke-69e6e73560f5826562c16b36/lava-logs.yaml 2026-04-21 03:13:09,476 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:13:09,590 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:13:09,592 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-smoke-69e6e73560f5826562c16b36/lava_callback.json.gz 2026-04-21 03:13:09,716 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:13:09,730 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6e73560f5826562c16b36 HTTP/1.1" 200 None 2026-04-21 03:13:09,732 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:13:09,745 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c21560f5826562c15091 HTTP/1.1" 200 None 2026-04-21 03:13:09,747 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:13:09,769 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6e73560f5826562c16b36 HTTP/1.1" 200 None 2026-04-21 03:13:09,889 - lava_callback - INFO - Completed processing callback for node 69e6e73560f5826562c16b36 INFO: 172.20.0.1:57344 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 03:13:15,842 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:13:15,860 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:57354 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48650 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:13:25,941 - lava_callback - INFO - Received callback for node 69e6e66f60f5826562c16aaa INFO: 172.20.0.1:48660 - "POST /node/69e6e66f60f5826562c16aaa HTTP/1.0" 202 Accepted 2026-04-21 03:13:25,947 - lava_callback - INFO - Starting async job submit for node 69e6e66f60f5826562c16aaa 2026-04-21 03:13:25,967 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:13:25,981 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6e66f60f5826562c16aaa HTTP/1.1" 200 None 2026-04-21 03:13:25,982 - lava_callback - DEBUG - Processing node 69e6e66f60f5826562c16aaa: baseline-arm64 2026-04-21 03:13:26,217 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:13:26,296 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:13:26,298 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6e66f60f5826562c16aaa/log.txt.gz 2026-04-21 03:13:26,421 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.20.0.1:48674 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:13:26,585 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:13:26,587 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6e66f60f5826562c16aaa/lava-logs.yaml 2026-04-21 03:13:26,594 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:13:26,719 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:13:26,721 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6e66f60f5826562c16aaa/lava_callback.json.gz DEBUG: Setup failed for 69e6e66f60f5826562c16aaa. Transitting job node result: incomplete -> fail 2026-04-21 03:13:26,767 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:13:26,783 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6e66f60f5826562c16aaa HTTP/1.1" 200 None 2026-04-21 03:13:26,784 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:13:26,803 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 03:13:26,805 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:13:26,825 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6e66f60f5826562c16aaa HTTP/1.1" 200 None 2026-04-21 03:13:26,883 - lava_callback - INFO - Completed processing callback for node 69e6e66f60f5826562c16aaa 2026-04-21 03:13:32,996 - lava_callback - INFO - Received callback for node 69e4484b3551964617339943 INFO: 172.20.0.1:38256 - "POST /node/69e4484b3551964617339943 HTTP/1.0" 202 Accepted 2026-04-21 03:13:33,002 - lava_callback - INFO - Starting async job submit for node 69e4484b3551964617339943 2026-04-21 03:13:33,003 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:13:33,020 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4484b3551964617339943 HTTP/1.1" 200 None 2026-04-21 03:13:33,021 - lava_callback - DEBUG - Processing node 69e4484b3551964617339943: baseline-arm64-baylibre 2026-04-21 03:13:33,021 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4484b3551964617339943 2026-04-21 03:13:33,022 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:13:33,036 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4484b3551964617339943 HTTP/1.1" 200 None 2026-04-21 03:13:33,037 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:13:33,054 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 03:13:33,056 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:13:33,073 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4484b3551964617339943 HTTP/1.1" 200 None 2026-04-21 03:13:33,074 - lava_callback - ERROR - Error processing callback for node 69e4484b3551964617339943: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:38262 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:13:45,862 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:13:45,877 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:38334 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60588 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55920 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51636 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51650 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40864 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40880 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:14:32,989 - lava_callback - INFO - Received callback for node 69e448963551964617339960 INFO: 172.20.0.1:40514 - "POST /node/69e448963551964617339960 HTTP/1.0" 202 Accepted 2026-04-21 03:14:32,994 - lava_callback - INFO - Starting async job submit for node 69e448963551964617339960 2026-04-21 03:14:32,995 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:14:33,011 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e448963551964617339960 HTTP/1.1" 200 None 2026-04-21 03:14:33,012 - lava_callback - DEBUG - Processing node 69e448963551964617339960: baseline-arm64-baylibre 2026-04-21 03:14:33,012 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e448963551964617339960 2026-04-21 03:14:33,013 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:14:33,025 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e448963551964617339960 HTTP/1.1" 200 None 2026-04-21 03:14:33,026 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:14:33,039 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 03:14:33,041 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:14:33,059 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e448963551964617339960 HTTP/1.1" 200 None 2026-04-21 03:14:33,060 - lava_callback - ERROR - Error processing callback for node 69e448963551964617339960: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:40526 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52752 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33742 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54722 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43514 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43526 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41088 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41100 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:15:35,872 - lava_callback - INFO - Received callback for node 69e6e6ea60f5826562c16b14 INFO: 172.20.0.1:57218 - "POST /node/69e6e6ea60f5826562c16b14 HTTP/1.0" 202 Accepted 2026-04-21 03:15:35,878 - lava_callback - INFO - Starting async job submit for node 69e6e6ea60f5826562c16b14 2026-04-21 03:15:35,901 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:15:35,917 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6e6ea60f5826562c16b14 HTTP/1.1" 200 None 2026-04-21 03:15:35,917 - lava_callback - DEBUG - Processing node 69e6e6ea60f5826562c16b14: baseline-nfs-arm64 2026-04-21 03:15:36,145 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:15:36,275 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:15:36,277 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6e6ea60f5826562c16b14/log.txt.gz 2026-04-21 03:15:36,408 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.20.0.1:57224 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:15:36,722 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:15:36,724 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6e6ea60f5826562c16b14/lava-logs.yaml 2026-04-21 03:15:36,731 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:15:36,858 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:15:36,859 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6e6ea60f5826562c16b14/lava_callback.json.gz DEBUG: Setup failed for 69e6e6ea60f5826562c16b14. Transitting job node result: incomplete -> fail 2026-04-21 03:15:36,902 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:15:36,929 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6e6ea60f5826562c16b14 HTTP/1.1" 200 None 2026-04-21 03:15:36,931 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:15:36,945 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 03:15:36,947 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:15:36,971 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6e6ea60f5826562c16b14 HTTP/1.1" 200 None 2026-04-21 03:15:37,043 - lava_callback - INFO - Completed processing callback for node 69e6e6ea60f5826562c16b14 2026-04-21 03:15:45,879 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:15:45,897 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:47278 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54344 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44666 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42374 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:42386 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42336 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42338 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33362 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56420 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37734 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37688 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:17:13,616 - lava_callback - INFO - Received callback for node 69e4492e35519646173399a7 INFO: 172.20.0.1:57826 - "POST /node/69e4492e35519646173399a7 HTTP/1.0" 202 Accepted 2026-04-21 03:17:13,623 - lava_callback - INFO - Starting async job submit for node 69e4492e35519646173399a7 2026-04-21 03:17:13,624 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:17:13,639 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4492e35519646173399a7 HTTP/1.1" 200 None 2026-04-21 03:17:13,640 - lava_callback - DEBUG - Processing node 69e4492e35519646173399a7: baseline-arm64-baylibre 2026-04-21 03:17:13,640 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4492e35519646173399a7 2026-04-21 03:17:13,641 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:17:13,658 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4492e35519646173399a7 HTTP/1.1" 200 None 2026-04-21 03:17:13,659 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:17:13,672 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 03:17:13,673 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:17:13,690 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4492e35519646173399a7 HTTP/1.1" 200 None 2026-04-21 03:17:13,691 - lava_callback - ERROR - Error processing callback for node 69e4492e35519646173399a7: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:57836 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57852 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55196 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55208 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37518 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38408 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43748 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52558 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57902 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57916 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57256 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:18:24,978 - lava_callback - INFO - Received callback for node 69e6ec0e60f5826562c16d73 INFO: 172.20.0.1:57270 - "POST /node/69e6ec0e60f5826562c16d73 HTTP/1.0" 202 Accepted 2026-04-21 03:18:24,984 - lava_callback - INFO - Starting async job submit for node 69e6ec0e60f5826562c16d73 2026-04-21 03:18:25,007 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:18:25,021 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6ec0e60f5826562c16d73 HTTP/1.1" 200 None 2026-04-21 03:18:25,022 - lava_callback - DEBUG - Processing node 69e6ec0e60f5826562c16d73: baseline-arm64-broonie 2026-04-21 03:18:25,329 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:18:25,456 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:18:25,458 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6ec0e60f5826562c16d73/log.txt.gz 2026-04-21 03:18:25,631 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:18:25,816 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:18:25,818 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6ec0e60f5826562c16d73/lava-logs.yaml 2026-04-21 03:18:25,826 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:18:25,925 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:18:25,927 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6ec0e60f5826562c16d73/lava_callback.json.gz 2026-04-21 03:18:25,959 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:18:25,973 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6ec0e60f5826562c16d73 HTTP/1.1" 200 None 2026-04-21 03:18:25,974 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:18:25,987 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6522047863bb428fe4af5 HTTP/1.1" 200 None 2026-04-21 03:18:25,989 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:18:26,012 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6ec0e60f5826562c16d73 HTTP/1.1" 200 None 2026-04-21 03:18:26,014 - lava_callback - INFO - Completed processing callback for node 69e6ec0e60f5826562c16d73 INFO: 172.20.0.1:57276 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:18:33,254 - lava_callback - INFO - Received callback for node 69e4497935519646173399d1 INFO: 172.20.0.1:39102 - "POST /node/69e4497935519646173399d1 HTTP/1.0" 202 Accepted 2026-04-21 03:18:33,260 - lava_callback - INFO - Starting async job submit for node 69e4497935519646173399d1 2026-04-21 03:18:33,261 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:18:33,281 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4497935519646173399d1 HTTP/1.1" 200 None 2026-04-21 03:18:33,282 - lava_callback - DEBUG - Processing node 69e4497935519646173399d1: baseline-arm64-baylibre 2026-04-21 03:18:33,282 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4497935519646173399d1 2026-04-21 03:18:33,283 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:18:33,296 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4497935519646173399d1 HTTP/1.1" 200 None 2026-04-21 03:18:33,297 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:18:33,309 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 03:18:33,311 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:18:33,328 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4497935519646173399d1 HTTP/1.1" 200 None 2026-04-21 03:18:33,329 - lava_callback - ERROR - Error processing callback for node 69e4497935519646173399d1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:39106 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:18:45,899 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:18:45,915 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:43384 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45876 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48652 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47480 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47486 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54404 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54410 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45004 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51432 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42712 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36586 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34026 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34034 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58416 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58422 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46404 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34246 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:20:53,398 - lava_callback - INFO - Received callback for node 69e44a043551964617339a13 INFO: 172.20.0.1:46040 - "POST /node/69e44a043551964617339a13 HTTP/1.0" 202 Accepted 2026-04-21 03:20:53,404 - lava_callback - INFO - Starting async job submit for node 69e44a043551964617339a13 2026-04-21 03:20:53,405 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:20:53,419 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44a043551964617339a13 HTTP/1.1" 200 None 2026-04-21 03:20:53,420 - lava_callback - DEBUG - Processing node 69e44a043551964617339a13: baseline-arm-baylibre 2026-04-21 03:20:53,420 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44a043551964617339a13 2026-04-21 03:20:53,420 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:20:53,433 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44a043551964617339a13 HTTP/1.1" 200 None 2026-04-21 03:20:53,435 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:20:53,447 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:20:53,449 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:20:53,473 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44a043551964617339a13 HTTP/1.1" 200 None 2026-04-21 03:20:53,474 - lava_callback - ERROR - Error processing callback for node 69e44a043551964617339a13: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:46054 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42692 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59162 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59174 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47922 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47926 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:21:33,392 - lava_callback - INFO - Received callback for node 69e44a283551964617339a28 INFO: 172.20.0.1:44702 - "POST /node/69e44a283551964617339a28 HTTP/1.0" 202 Accepted 2026-04-21 03:21:33,397 - lava_callback - INFO - Starting async job submit for node 69e44a283551964617339a28 2026-04-21 03:21:33,399 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:21:33,413 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44a283551964617339a28 HTTP/1.1" 200 None 2026-04-21 03:21:33,413 - lava_callback - DEBUG - Processing node 69e44a283551964617339a28: baseline-arm-baylibre 2026-04-21 03:21:33,414 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44a283551964617339a28 2026-04-21 03:21:33,414 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:21:33,426 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44a283551964617339a28 HTTP/1.1" 200 None 2026-04-21 03:21:33,428 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:21:33,440 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:21:33,442 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:21:33,462 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44a283551964617339a28 HTTP/1.1" 200 None 2026-04-21 03:21:33,463 - lava_callback - ERROR - Error processing callback for node 69e44a283551964617339a28: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:44718 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35562 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:21:53,393 - lava_callback - INFO - Received callback for node 69e44a4c3551964617339a35 INFO: 172.20.0.1:38734 - "POST /node/69e44a4c3551964617339a35 HTTP/1.0" 202 Accepted 2026-04-21 03:21:53,399 - lava_callback - INFO - Starting async job submit for node 69e44a4c3551964617339a35 2026-04-21 03:21:53,400 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:21:53,414 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44a4c3551964617339a35 HTTP/1.1" 200 None 2026-04-21 03:21:53,415 - lava_callback - DEBUG - Processing node 69e44a4c3551964617339a35: baseline-arm-baylibre 2026-04-21 03:21:53,415 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44a4c3551964617339a35 2026-04-21 03:21:53,416 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:21:53,429 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44a4c3551964617339a35 HTTP/1.1" 200 None 2026-04-21 03:21:53,430 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:21:53,468 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:21:53,470 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:21:53,526 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44a4c3551964617339a35 HTTP/1.1" 200 None 2026-04-21 03:21:53,527 - lava_callback - ERROR - Error processing callback for node 69e44a4c3551964617339a35: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:38742 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42268 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45288 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45290 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36688 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36694 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45060 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37444 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:22:53,496 - lava_callback - INFO - Received callback for node 69e44a7c3551964617339a44 INFO: 172.20.0.1:49604 - "POST /node/69e44a7c3551964617339a44 HTTP/1.0" 202 Accepted 2026-04-21 03:22:53,501 - lava_callback - INFO - Starting async job submit for node 69e44a7c3551964617339a44 2026-04-21 03:22:53,502 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:22:53,529 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44a7c3551964617339a44 HTTP/1.1" 200 None 2026-04-21 03:22:53,530 - lava_callback - DEBUG - Processing node 69e44a7c3551964617339a44: baseline-arm64-baylibre 2026-04-21 03:22:53,530 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44a7c3551964617339a44 2026-04-21 03:22:53,531 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:22:53,544 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44a7c3551964617339a44 HTTP/1.1" 200 None 2026-04-21 03:22:53,546 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:22:53,590 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 03:22:53,592 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:22:53,617 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44a7c3551964617339a44 HTTP/1.1" 200 None 2026-04-21 03:22:53,618 - lava_callback - ERROR - Error processing callback for node 69e44a7c3551964617339a44: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49608 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51570 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49088 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49092 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40864 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40868 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47562 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38934 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58050 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56470 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33888 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33892 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44818 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44834 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45382 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43600 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:24:53,605 - lava_callback - INFO - Received callback for node 69e44b023551964617339a91 INFO: 172.20.0.1:37634 - "POST /node/69e44b023551964617339a91 HTTP/1.0" 202 Accepted 2026-04-21 03:24:53,611 - lava_callback - INFO - Starting async job submit for node 69e44b023551964617339a91 2026-04-21 03:24:53,612 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:24:53,627 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44b023551964617339a91 HTTP/1.1" 200 None 2026-04-21 03:24:53,628 - lava_callback - DEBUG - Processing node 69e44b023551964617339a91: baseline-arm-baylibre 2026-04-21 03:24:53,628 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44b023551964617339a91 2026-04-21 03:24:53,629 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:24:53,643 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44b023551964617339a91 HTTP/1.1" 200 None 2026-04-21 03:24:53,645 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:24:53,663 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:24:53,665 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:24:53,688 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44b023551964617339a91 HTTP/1.1" 200 None 2026-04-21 03:24:53,689 - lava_callback - ERROR - Error processing callback for node 69e44b023551964617339a91: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:37646 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33662 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44974 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44980 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35908 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35920 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:25:33,623 - lava_callback - INFO - Received callback for node 69e44b263551964617339aa0 INFO: 172.20.0.1:42198 - "POST /node/69e44b263551964617339aa0 HTTP/1.0" 202 Accepted 2026-04-21 03:25:33,629 - lava_callback - INFO - Starting async job submit for node 69e44b263551964617339aa0 2026-04-21 03:25:33,630 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:25:33,645 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44b263551964617339aa0 HTTP/1.1" 200 None 2026-04-21 03:25:33,646 - lava_callback - DEBUG - Processing node 69e44b263551964617339aa0: baseline-arm-baylibre 2026-04-21 03:25:33,646 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44b263551964617339aa0 2026-04-21 03:25:33,646 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:25:33,661 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44b263551964617339aa0 HTTP/1.1" 200 None 2026-04-21 03:25:33,663 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:25:33,676 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:25:33,677 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:25:33,700 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44b263551964617339aa0 HTTP/1.1" 200 None 2026-04-21 03:25:33,701 - lava_callback - ERROR - Error processing callback for node 69e44b263551964617339aa0: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:42210 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53106 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54966 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56710 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:26:13,659 - lava_callback - INFO - Received callback for node 69e44b493551964617339aae INFO: 172.20.0.1:39056 - "POST /node/69e44b493551964617339aae HTTP/1.0" 202 Accepted 2026-04-21 03:26:13,665 - lava_callback - INFO - Starting async job submit for node 69e44b493551964617339aae 2026-04-21 03:26:13,665 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:26:13,680 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44b493551964617339aae HTTP/1.1" 200 None 2026-04-21 03:26:13,681 - lava_callback - DEBUG - Processing node 69e44b493551964617339aae: baseline-arm-baylibre 2026-04-21 03:26:13,682 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44b493551964617339aae 2026-04-21 03:26:13,682 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:26:13,698 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44b493551964617339aae HTTP/1.1" 200 None 2026-04-21 03:26:13,699 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:26:13,713 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:26:13,714 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:26:13,731 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44b493551964617339aae HTTP/1.1" 200 None 2026-04-21 03:26:13,732 - lava_callback - ERROR - Error processing callback for node 69e44b493551964617339aae: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:39060 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39062 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42714 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42726 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39048 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44210 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:26:53,678 - lava_callback - INFO - Received callback for node 69e44b6d3551964617339ab9 INFO: 172.20.0.1:47396 - "POST /node/69e44b6d3551964617339ab9 HTTP/1.0" 202 Accepted 2026-04-21 03:26:53,688 - lava_callback - INFO - Starting async job submit for node 69e44b6d3551964617339ab9 2026-04-21 03:26:53,689 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:26:53,710 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44b6d3551964617339ab9 HTTP/1.1" 200 None 2026-04-21 03:26:53,711 - lava_callback - DEBUG - Processing node 69e44b6d3551964617339ab9: baseline-arm-baylibre 2026-04-21 03:26:53,711 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44b6d3551964617339ab9 2026-04-21 03:26:53,712 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:26:53,727 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44b6d3551964617339ab9 HTTP/1.1" 200 None 2026-04-21 03:26:53,728 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:26:53,743 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:26:53,744 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:26:53,763 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44b6d3551964617339ab9 HTTP/1.1" 200 None 2026-04-21 03:26:53,765 - lava_callback - ERROR - Error processing callback for node 69e44b6d3551964617339ab9: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47412 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36226 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45432 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45436 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49002 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49006 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:27:33,791 - lava_callback - INFO - Received callback for node 69e44b913551964617339aca INFO: 172.20.0.1:39044 - "POST /node/69e44b913551964617339aca HTTP/1.0" 202 Accepted 2026-04-21 03:27:33,797 - lava_callback - INFO - Starting async job submit for node 69e44b913551964617339aca 2026-04-21 03:27:33,798 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:27:33,812 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44b913551964617339aca HTTP/1.1" 200 None 2026-04-21 03:27:33,813 - lava_callback - DEBUG - Processing node 69e44b913551964617339aca: baseline-arm-baylibre 2026-04-21 03:27:33,813 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44b913551964617339aca 2026-04-21 03:27:33,814 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:27:33,830 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44b913551964617339aca HTTP/1.1" 200 None 2026-04-21 03:27:33,831 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:27:33,845 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:27:33,847 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:27:33,865 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44b913551964617339aca HTTP/1.1" 200 None 2026-04-21 03:27:33,866 - lava_callback - ERROR - Error processing callback for node 69e44b913551964617339aca: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:39060 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47176 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55978 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58322 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38446 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:38452 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34452 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34462 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39668 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60688 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54584 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37114 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:29:13,860 - lava_callback - INFO - Received callback for node 69e44c053551964617339b03 INFO: 172.20.0.1:58208 - "POST /node/69e44c053551964617339b03 HTTP/1.0" 202 Accepted 2026-04-21 03:29:13,865 - lava_callback - INFO - Starting async job submit for node 69e44c053551964617339b03 2026-04-21 03:29:13,866 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:29:13,906 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44c053551964617339b03 HTTP/1.1" 200 None 2026-04-21 03:29:13,907 - lava_callback - DEBUG - Processing node 69e44c053551964617339b03: baseline-arm-baylibre 2026-04-21 03:29:13,907 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44c053551964617339b03 2026-04-21 03:29:13,908 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:29:13,921 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44c053551964617339b03 HTTP/1.1" 200 None 2026-04-21 03:29:13,922 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:29:13,936 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:29:13,937 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:29:13,954 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44c053551964617339b03 HTTP/1.1" 200 None 2026-04-21 03:29:13,955 - lava_callback - ERROR - Error processing callback for node 69e44c053551964617339b03: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:58210 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58216 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53280 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53286 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52320 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37418 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:29:53,880 - lava_callback - INFO - Received callback for node 69e44c293551964617339b0e INFO: 172.20.0.1:53608 - "POST /node/69e44c293551964617339b0e HTTP/1.0" 202 Accepted 2026-04-21 03:29:53,886 - lava_callback - INFO - Starting async job submit for node 69e44c293551964617339b0e 2026-04-21 03:29:53,887 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:29:53,902 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44c293551964617339b0e HTTP/1.1" 200 None 2026-04-21 03:29:53,902 - lava_callback - DEBUG - Processing node 69e44c293551964617339b0e: baseline-arm-baylibre 2026-04-21 03:29:53,903 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44c293551964617339b0e 2026-04-21 03:29:53,903 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:29:53,916 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44c293551964617339b0e HTTP/1.1" 200 None 2026-04-21 03:29:53,917 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:29:53,930 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:29:53,931 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:29:53,957 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44c293551964617339b0e HTTP/1.1" 200 None 2026-04-21 03:29:53,958 - lava_callback - ERROR - Error processing callback for node 69e44c293551964617339b0e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:53616 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37254 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48328 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48342 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33198 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33202 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49166 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:30:39,378 - lava_callback - INFO - Received callback for node 69e44c5c3551964617339b1e INFO: 172.20.0.1:49180 - "POST /node/69e44c5c3551964617339b1e HTTP/1.0" 202 Accepted 2026-04-21 03:30:39,384 - lava_callback - INFO - Starting async job submit for node 69e44c5c3551964617339b1e 2026-04-21 03:30:39,385 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:30:39,398 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44c5c3551964617339b1e HTTP/1.1" 200 None 2026-04-21 03:30:39,399 - lava_callback - DEBUG - Processing node 69e44c5c3551964617339b1e: baseline-arm64-clabbe 2026-04-21 03:30:39,399 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44c5c3551964617339b1e 2026-04-21 03:30:39,400 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:30:39,413 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44c5c3551964617339b1e HTTP/1.1" 200 None 2026-04-21 03:30:39,415 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:30:39,428 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 03:30:39,429 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:30:39,447 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44c5c3551964617339b1e HTTP/1.1" 200 None 2026-04-21 03:30:39,448 - lava_callback - ERROR - Error processing callback for node 69e44c5c3551964617339b1e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:40846 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40824 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42676 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41980 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41996 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57690 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57702 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40788 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33910 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:31:54,072 - lava_callback - INFO - Received callback for node 69e44c983551964617339b42 INFO: 172.20.0.1:35822 - "POST /node/69e44c983551964617339b42 HTTP/1.0" 202 Accepted 2026-04-21 03:31:54,078 - lava_callback - INFO - Starting async job submit for node 69e44c983551964617339b42 2026-04-21 03:31:54,079 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:31:54,092 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44c983551964617339b42 HTTP/1.1" 200 None 2026-04-21 03:31:54,093 - lava_callback - DEBUG - Processing node 69e44c983551964617339b42: baseline-arm-baylibre 2026-04-21 03:31:54,093 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44c983551964617339b42 2026-04-21 03:31:54,094 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:31:54,108 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44c983551964617339b42 HTTP/1.1" 200 None 2026-04-21 03:31:54,110 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:31:54,133 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:31:54,135 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:31:54,154 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44c983551964617339b42 HTTP/1.1" 200 None 2026-04-21 03:31:54,155 - lava_callback - ERROR - Error processing callback for node 69e44c983551964617339b42: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:35830 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60228 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36490 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:36496 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33912 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33920 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:32:33,980 - lava_callback - INFO - Received callback for node 69e44cbd3551964617339b56 INFO: 172.20.0.1:57676 - "POST /node/69e44cbd3551964617339b56 HTTP/1.0" 202 Accepted 2026-04-21 03:32:33,985 - lava_callback - INFO - Starting async job submit for node 69e44cbd3551964617339b56 2026-04-21 03:32:33,986 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:32:34,000 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44cbd3551964617339b56 HTTP/1.1" 200 None 2026-04-21 03:32:34,001 - lava_callback - DEBUG - Processing node 69e44cbd3551964617339b56: baseline-arm-baylibre 2026-04-21 03:32:34,001 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44cbd3551964617339b56 2026-04-21 03:32:34,002 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:32:34,018 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44cbd3551964617339b56 HTTP/1.1" 200 None 2026-04-21 03:32:34,019 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:32:34,032 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:32:34,033 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:32:34,050 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44cbd3551964617339b56 HTTP/1.1" 200 None 2026-04-21 03:32:34,051 - lava_callback - ERROR - Error processing callback for node 69e44cbd3551964617339b56: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:57684 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36668 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50922 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:32:59,462 - lava_callback - INFO - Received callback for node 69e44ce43551964617339b5f INFO: 172.20.0.1:50930 - "POST /node/69e44ce43551964617339b5f HTTP/1.0" 202 Accepted 2026-04-21 03:32:59,467 - lava_callback - INFO - Starting async job submit for node 69e44ce43551964617339b5f 2026-04-21 03:32:59,468 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:32:59,484 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44ce43551964617339b5f HTTP/1.1" 200 None 2026-04-21 03:32:59,485 - lava_callback - DEBUG - Processing node 69e44ce43551964617339b5f: baseline-arm-clabbe 2026-04-21 03:32:59,485 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44ce43551964617339b5f 2026-04-21 03:32:59,485 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:32:59,498 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44ce43551964617339b5f HTTP/1.1" 200 None 2026-04-21 03:32:59,500 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:32:59,515 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:32:59,517 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:32:59,534 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44ce43551964617339b5f HTTP/1.1" 200 None 2026-04-21 03:32:59,535 - lava_callback - ERROR - Error processing callback for node 69e44ce43551964617339b5f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:33544 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47888 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47894 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41590 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41594 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40120 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:33:39,449 - lava_callback - INFO - Received callback for node 69e44d0b3551964617339b6c INFO: 172.20.0.1:40136 - "POST /node/69e44d0b3551964617339b6c HTTP/1.0" 202 Accepted 2026-04-21 03:33:39,456 - lava_callback - INFO - Starting async job submit for node 69e44d0b3551964617339b6c 2026-04-21 03:33:39,457 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:33:39,482 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44d0b3551964617339b6c HTTP/1.1" 200 None 2026-04-21 03:33:39,483 - lava_callback - DEBUG - Processing node 69e44d0b3551964617339b6c: baseline-arm-clabbe 2026-04-21 03:33:39,483 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44d0b3551964617339b6c 2026-04-21 03:33:39,484 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:33:39,504 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44d0b3551964617339b6c HTTP/1.1" 200 None 2026-04-21 03:33:39,506 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:33:39,519 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:33:39,520 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:33:39,540 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44d0b3551964617339b6c HTTP/1.1" 200 None 2026-04-21 03:33:39,541 - lava_callback - ERROR - Error processing callback for node 69e44d0b3551964617339b6c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:56894 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47924 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42808 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35980 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35992 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:34:19,501 - lava_callback - INFO - Received callback for node 69e44d323551964617339b7e INFO: 172.20.0.1:35998 - "POST /node/69e44d323551964617339b7e HTTP/1.0" 202 Accepted 2026-04-21 03:34:19,506 - lava_callback - INFO - Starting async job submit for node 69e44d323551964617339b7e 2026-04-21 03:34:19,507 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:34:19,523 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44d323551964617339b7e HTTP/1.1" 200 None 2026-04-21 03:34:19,524 - lava_callback - DEBUG - Processing node 69e44d323551964617339b7e: baseline-arm-clabbe 2026-04-21 03:34:19,524 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44d323551964617339b7e 2026-04-21 03:34:19,525 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:34:19,547 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44d323551964617339b7e HTTP/1.1" 200 None 2026-04-21 03:34:19,549 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:34:19,561 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:34:19,563 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:34:19,588 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44d323551964617339b7e HTTP/1.1" 200 None 2026-04-21 03:34:19,589 - lava_callback - ERROR - Error processing callback for node 69e44d323551964617339b7e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:44760 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44766 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57458 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53150 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56750 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:34:59,504 - lava_callback - INFO - Received callback for node 69e44d583551964617339b89 INFO: 172.20.0.1:56760 - "POST /node/69e44d583551964617339b89 HTTP/1.0" 202 Accepted 2026-04-21 03:34:59,509 - lava_callback - INFO - Starting async job submit for node 69e44d583551964617339b89 2026-04-21 03:34:59,510 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:34:59,524 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44d583551964617339b89 HTTP/1.1" 200 None 2026-04-21 03:34:59,525 - lava_callback - DEBUG - Processing node 69e44d583551964617339b89: baseline-arm-clabbe 2026-04-21 03:34:59,525 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44d583551964617339b89 2026-04-21 03:34:59,526 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:34:59,538 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44d583551964617339b89 HTTP/1.1" 200 None 2026-04-21 03:34:59,540 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:34:59,553 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:34:59,554 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:34:59,572 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44d583551964617339b89 HTTP/1.1" 200 None 2026-04-21 03:34:59,573 - lava_callback - ERROR - Error processing callback for node 69e44d583551964617339b89: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:46246 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37874 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37884 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46230 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46244 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:35:35,353 - lava_callback - INFO - Received callback for node 69e6ef9c60f5826562c16ef4 INFO: 172.20.0.1:45382 - "POST /node/69e6ef9c60f5826562c16ef4 HTTP/1.0" 202 Accepted 2026-04-21 03:35:35,360 - lava_callback - INFO - Starting async job submit for node 69e6ef9c60f5826562c16ef4 2026-04-21 03:35:35,384 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:35:35,397 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6ef9c60f5826562c16ef4 HTTP/1.1" 200 None 2026-04-21 03:35:35,398 - lava_callback - DEBUG - Processing node 69e6ef9c60f5826562c16ef4: baseline-nfs-arm64 2026-04-21 03:35:35,930 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:35:36,201 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:35:36,203 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6ef9c60f5826562c16ef4/log.txt.gz 2026-04-21 03:35:36,482 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.20.0.1:45394 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:35:36,816 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:35:36,818 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6ef9c60f5826562c16ef4/lava-logs.yaml 2026-04-21 03:35:36,829 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 03:35:36,978 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 03:35:36,980 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6ef9c60f5826562c16ef4/lava_callback.json.gz 2026-04-21 03:35:37,011 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:35:37,024 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6ef9c60f5826562c16ef4 HTTP/1.1" 200 None 2026-04-21 03:35:37,026 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:35:37,040 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 03:35:37,041 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:35:37,072 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6ef9c60f5826562c16ef4 HTTP/1.1" 200 None 2026-04-21 03:35:37,074 - lava_callback - INFO - Completed processing callback for node 69e6ef9c60f5826562c16ef4 2026-04-21 03:35:39,588 - lava_callback - INFO - Received callback for node 69e44d7f3551964617339b97 INFO: 172.20.0.1:45410 - "POST /node/69e44d7f3551964617339b97 HTTP/1.0" 202 Accepted 2026-04-21 03:35:39,593 - lava_callback - INFO - Starting async job submit for node 69e44d7f3551964617339b97 2026-04-21 03:35:39,594 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:35:39,608 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44d7f3551964617339b97 HTTP/1.1" 200 None 2026-04-21 03:35:39,608 - lava_callback - DEBUG - Processing node 69e44d7f3551964617339b97: baseline-arm-clabbe 2026-04-21 03:35:39,609 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44d7f3551964617339b97 2026-04-21 03:35:39,609 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:35:39,622 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44d7f3551964617339b97 HTTP/1.1" 200 None 2026-04-21 03:35:39,623 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:35:39,636 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:35:39,637 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:35:39,654 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44d7f3551964617339b97 HTTP/1.1" 200 None 2026-04-21 03:35:39,655 - lava_callback - ERROR - Error processing callback for node 69e44d7f3551964617339b97: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-04-21 03:35:45,920 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:35:45,936 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:40438 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58024 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42572 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42826 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:42832 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51482 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51498 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56942 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:36:39,656 - lava_callback - INFO - Received callback for node 69e44db23551964617339bab INFO: 172.20.0.1:56958 - "POST /node/69e44db23551964617339bab HTTP/1.0" 202 Accepted 2026-04-21 03:36:39,662 - lava_callback - INFO - Starting async job submit for node 69e44db23551964617339bab 2026-04-21 03:36:39,663 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:36:39,677 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44db23551964617339bab HTTP/1.1" 200 None 2026-04-21 03:36:39,678 - lava_callback - DEBUG - Processing node 69e44db23551964617339bab: baseline-arm64-clabbe 2026-04-21 03:36:39,678 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44db23551964617339bab 2026-04-21 03:36:39,678 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:36:39,691 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44db23551964617339bab HTTP/1.1" 200 None 2026-04-21 03:36:39,692 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:36:39,705 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 03:36:39,707 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:36:39,724 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44db23551964617339bab HTTP/1.1" 200 None 2026-04-21 03:36:39,725 - lava_callback - ERROR - Error processing callback for node 69e44db23551964617339bab: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:44588 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37664 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47130 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38594 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:38606 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37172 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37178 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40126 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35102 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44600 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35874 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51782 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51792 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51550 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51556 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46232 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:38:39,740 - lava_callback - INFO - Received callback for node 69e44e393551964617339be0 INFO: 172.20.0.1:36514 - "POST /node/69e44e393551964617339be0 HTTP/1.0" 202 Accepted 2026-04-21 03:38:39,746 - lava_callback - INFO - Starting async job submit for node 69e44e393551964617339be0 2026-04-21 03:38:39,747 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:38:39,762 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44e393551964617339be0 HTTP/1.1" 200 None 2026-04-21 03:38:39,762 - lava_callback - DEBUG - Processing node 69e44e393551964617339be0: baseline-arm-clabbe 2026-04-21 03:38:39,763 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44e393551964617339be0 2026-04-21 03:38:39,763 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:38:39,776 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44e393551964617339be0 HTTP/1.1" 200 None 2026-04-21 03:38:39,777 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:38:39,790 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:38:39,792 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:38:39,814 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44e393551964617339be0 HTTP/1.1" 200 None 2026-04-21 03:38:39,815 - lava_callback - ERROR - Error processing callback for node 69e44e393551964617339be0: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:36520 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41206 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33612 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47300 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47310 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:39:19,736 - lava_callback - INFO - Received callback for node 69e44e603551964617339bf2 INFO: 172.20.0.1:45096 - "POST /node/69e44e603551964617339bf2 HTTP/1.0" 202 Accepted 2026-04-21 03:39:19,742 - lava_callback - INFO - Starting async job submit for node 69e44e603551964617339bf2 2026-04-21 03:39:19,743 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:39:19,757 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44e603551964617339bf2 HTTP/1.1" 200 None 2026-04-21 03:39:19,758 - lava_callback - DEBUG - Processing node 69e44e603551964617339bf2: baseline-arm-clabbe 2026-04-21 03:39:19,758 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44e603551964617339bf2 2026-04-21 03:39:19,759 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:39:19,777 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44e603551964617339bf2 HTTP/1.1" 200 None 2026-04-21 03:39:19,778 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:39:19,796 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:39:19,798 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:39:19,816 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44e603551964617339bf2 HTTP/1.1" 200 None 2026-04-21 03:39:19,817 - lava_callback - ERROR - Error processing callback for node 69e44e603551964617339bf2: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:45102 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45114 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59998 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41956 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59704 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50470 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50872 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50882 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:40:19,856 - lava_callback - INFO - Received callback for node 69e44e923551964617339bfe INFO: 172.20.0.1:58544 - "POST /node/69e44e923551964617339bfe HTTP/1.0" 202 Accepted 2026-04-21 03:40:19,861 - lava_callback - INFO - Starting async job submit for node 69e44e923551964617339bfe 2026-04-21 03:40:19,863 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:40:19,877 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44e923551964617339bfe HTTP/1.1" 200 None 2026-04-21 03:40:19,877 - lava_callback - DEBUG - Processing node 69e44e923551964617339bfe: baseline-arm64-clabbe 2026-04-21 03:40:19,878 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44e923551964617339bfe 2026-04-21 03:40:19,878 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:40:19,893 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44e923551964617339bfe HTTP/1.1" 200 None 2026-04-21 03:40:19,894 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:40:19,906 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 03:40:19,908 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:40:19,926 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44e923551964617339bfe HTTP/1.1" 200 None 2026-04-21 03:40:19,927 - lava_callback - ERROR - Error processing callback for node 69e44e923551964617339bfe: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:58550 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58556 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36142 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39734 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46936 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44776 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39108 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39110 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:41:19,834 - lava_callback - INFO - Received callback for node 69e44ed03551964617339c1d INFO: 172.20.0.1:50346 - "POST /node/69e44ed03551964617339c1d HTTP/1.0" 202 Accepted 2026-04-21 03:41:19,839 - lava_callback - INFO - Starting async job submit for node 69e44ed03551964617339c1d 2026-04-21 03:41:19,841 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:41:19,854 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44ed03551964617339c1d HTTP/1.1" 200 None 2026-04-21 03:41:19,855 - lava_callback - DEBUG - Processing node 69e44ed03551964617339c1d: baseline-arm-clabbe 2026-04-21 03:41:19,856 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44ed03551964617339c1d 2026-04-21 03:41:19,856 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:41:19,869 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44ed03551964617339c1d HTTP/1.1" 200 None 2026-04-21 03:41:19,871 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:41:19,883 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:41:19,885 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:41:19,902 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44ed03551964617339c1d HTTP/1.1" 200 None 2026-04-21 03:41:19,903 - lava_callback - ERROR - Error processing callback for node 69e44ed03551964617339c1d: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:50352 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50362 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44586 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56830 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58806 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:41:59,811 - lava_callback - INFO - Received callback for node 69e44ef53551964617339c2c INFO: 172.20.0.1:46130 - "POST /node/69e44ef53551964617339c2c HTTP/1.0" 202 Accepted 2026-04-21 03:41:59,817 - lava_callback - INFO - Starting async job submit for node 69e44ef53551964617339c2c 2026-04-21 03:41:59,818 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:41:59,837 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44ef53551964617339c2c HTTP/1.1" 200 None 2026-04-21 03:41:59,838 - lava_callback - DEBUG - Processing node 69e44ef53551964617339c2c: baseline-arm-clabbe 2026-04-21 03:41:59,838 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44ef53551964617339c2c 2026-04-21 03:41:59,838 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:41:59,852 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44ef53551964617339c2c HTTP/1.1" 200 None 2026-04-21 03:41:59,854 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:41:59,866 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:41:59,867 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:41:59,885 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44ef53551964617339c2c HTTP/1.1" 200 None 2026-04-21 03:41:59,886 - lava_callback - ERROR - Error processing callback for node 69e44ef53551964617339c2c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:46144 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40360 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40372 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47952 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47968 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46318 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:42:39,911 - lava_callback - INFO - Received callback for node 69e44f1a3551964617339c35 INFO: 172.20.0.1:54170 - "POST /node/69e44f1a3551964617339c35 HTTP/1.0" 202 Accepted 2026-04-21 03:42:39,917 - lava_callback - INFO - Starting async job submit for node 69e44f1a3551964617339c35 2026-04-21 03:42:39,918 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:42:39,932 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44f1a3551964617339c35 HTTP/1.1" 200 None 2026-04-21 03:42:39,933 - lava_callback - DEBUG - Processing node 69e44f1a3551964617339c35: baseline-arm-clabbe 2026-04-21 03:42:39,933 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44f1a3551964617339c35 2026-04-21 03:42:39,933 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:42:39,946 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44f1a3551964617339c35 HTTP/1.1" 200 None 2026-04-21 03:42:39,948 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:42:39,961 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:42:39,962 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:42:39,980 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44f1a3551964617339c35 HTTP/1.1" 200 None 2026-04-21 03:42:39,981 - lava_callback - ERROR - Error processing callback for node 69e44f1a3551964617339c35: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54180 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37676 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36842 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:43:14,760 - lava_callback - INFO - Received callback for node 69e44f443551964617339c40 INFO: 172.20.0.1:60072 - "POST /node/69e44f443551964617339c40 HTTP/1.0" 202 Accepted 2026-04-21 03:43:14,765 - lava_callback - INFO - Starting async job submit for node 69e44f443551964617339c40 2026-04-21 03:43:14,767 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:43:14,782 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44f443551964617339c40 HTTP/1.1" 200 None 2026-04-21 03:43:14,783 - lava_callback - DEBUG - Processing node 69e44f443551964617339c40: baseline-arm64-baylibre 2026-04-21 03:43:14,783 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44f443551964617339c40 2026-04-21 03:43:14,783 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:43:14,883 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44f443551964617339c40 HTTP/1.1" 200 None 2026-04-21 03:43:14,885 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:43:14,903 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e33e363745d1193f227399 HTTP/1.1" 200 None 2026-04-21 03:43:14,904 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:43:14,923 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44f443551964617339c40 HTTP/1.1" 200 None 2026-04-21 03:43:14,924 - lava_callback - ERROR - Error processing callback for node 69e44f443551964617339c40: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:60082 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60084 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35308 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35312 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60760 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60642 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58244 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56782 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41054 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41068 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:44:20,099 - lava_callback - INFO - Received callback for node 69e44f883551964617339c5c INFO: 172.20.0.1:41598 - "POST /node/69e44f883551964617339c5c HTTP/1.0" 202 Accepted 2026-04-21 03:44:20,105 - lava_callback - INFO - Starting async job submit for node 69e44f883551964617339c5c 2026-04-21 03:44:20,106 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:44:20,120 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44f883551964617339c5c HTTP/1.1" 200 None 2026-04-21 03:44:20,120 - lava_callback - DEBUG - Processing node 69e44f883551964617339c5c: baseline-arm-clabbe 2026-04-21 03:44:20,121 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44f883551964617339c5c 2026-04-21 03:44:20,121 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:44:20,134 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44f883551964617339c5c HTTP/1.1" 200 None 2026-04-21 03:44:20,135 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:44:20,158 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:44:20,160 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:44:20,177 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44f883551964617339c5c HTTP/1.1" 200 None 2026-04-21 03:44:20,178 - lava_callback - ERROR - Error processing callback for node 69e44f883551964617339c5c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:41610 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41624 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33046 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44620 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51352 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:45:00,021 - lava_callback - INFO - Received callback for node 69e44fad3551964617339c6b INFO: 172.20.0.1:59542 - "POST /node/69e44fad3551964617339c6b HTTP/1.0" 202 Accepted 2026-04-21 03:45:00,027 - lava_callback - INFO - Starting async job submit for node 69e44fad3551964617339c6b 2026-04-21 03:45:00,028 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:45:00,044 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44fad3551964617339c6b HTTP/1.1" 200 None 2026-04-21 03:45:00,045 - lava_callback - DEBUG - Processing node 69e44fad3551964617339c6b: baseline-arm-clabbe 2026-04-21 03:45:00,045 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44fad3551964617339c6b 2026-04-21 03:45:00,046 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:45:00,062 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44fad3551964617339c6b HTTP/1.1" 200 None 2026-04-21 03:45:00,063 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:45:00,077 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:45:00,079 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:45:00,110 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44fad3551964617339c6b HTTP/1.1" 200 None 2026-04-21 03:45:00,112 - lava_callback - ERROR - Error processing callback for node 69e44fad3551964617339c6b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:59548 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45468 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45482 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59682 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59692 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51908 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:45:40,035 - lava_callback - INFO - Received callback for node 69e44fd33551964617339c74 INFO: 172.20.0.1:54794 - "POST /node/69e44fd33551964617339c74 HTTP/1.0" 202 Accepted 2026-04-21 03:45:40,041 - lava_callback - INFO - Starting async job submit for node 69e44fd33551964617339c74 2026-04-21 03:45:40,042 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:45:40,057 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44fd33551964617339c74 HTTP/1.1" 200 None 2026-04-21 03:45:40,058 - lava_callback - DEBUG - Processing node 69e44fd33551964617339c74: baseline-arm-clabbe 2026-04-21 03:45:40,059 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44fd33551964617339c74 2026-04-21 03:45:40,059 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:45:40,072 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44fd33551964617339c74 HTTP/1.1" 200 None 2026-04-21 03:45:40,074 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:45:40,087 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:45:40,088 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:45:40,111 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44fd33551964617339c74 HTTP/1.1" 200 None 2026-04-21 03:45:40,112 - lava_callback - ERROR - Error processing callback for node 69e44fd33551964617339c74: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54806 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59938 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49540 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49172 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49174 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:46:20,068 - lava_callback - INFO - Received callback for node 69e44ff83551964617339c7f INFO: 172.20.0.1:43586 - "POST /node/69e44ff83551964617339c7f HTTP/1.0" 202 Accepted 2026-04-21 03:46:20,074 - lava_callback - INFO - Starting async job submit for node 69e44ff83551964617339c7f 2026-04-21 03:46:20,075 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:46:20,089 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44ff83551964617339c7f HTTP/1.1" 200 None 2026-04-21 03:46:20,089 - lava_callback - DEBUG - Processing node 69e44ff83551964617339c7f: baseline-arm-clabbe 2026-04-21 03:46:20,090 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e44ff83551964617339c7f 2026-04-21 03:46:20,090 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:46:20,103 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e44ff83551964617339c7f HTTP/1.1" 200 None 2026-04-21 03:46:20,104 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:46:20,117 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:46:20,118 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:46:20,135 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e44ff83551964617339c7f HTTP/1.1" 200 None 2026-04-21 03:46:20,136 - lava_callback - ERROR - Error processing callback for node 69e44ff83551964617339c7f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:43588 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43590 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47208 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39514 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53876 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:47:00,117 - lava_callback - INFO - Received callback for node 69e4501e3551964617339c8a INFO: 172.20.0.1:47528 - "POST /node/69e4501e3551964617339c8a HTTP/1.0" 202 Accepted 2026-04-21 03:47:00,123 - lava_callback - INFO - Starting async job submit for node 69e4501e3551964617339c8a 2026-04-21 03:47:00,124 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:47:00,158 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4501e3551964617339c8a HTTP/1.1" 200 None 2026-04-21 03:47:00,159 - lava_callback - DEBUG - Processing node 69e4501e3551964617339c8a: baseline-arm-clabbe 2026-04-21 03:47:00,159 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4501e3551964617339c8a 2026-04-21 03:47:00,160 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:47:00,198 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4501e3551964617339c8a HTTP/1.1" 200 None 2026-04-21 03:47:00,200 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:47:00,213 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:47:00,215 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:47:00,252 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4501e3551964617339c8a HTTP/1.1" 200 None 2026-04-21 03:47:00,254 - lava_callback - ERROR - Error processing callback for node 69e4501e3551964617339c8a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47540 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33710 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33726 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:47:20,118 - lava_callback - INFO - Received callback for node 69e450423551964617339c97 INFO: 172.20.0.1:54162 - "POST /node/69e450423551964617339c97 HTTP/1.0" 202 Accepted 2026-04-21 03:47:20,126 - lava_callback - INFO - Starting async job submit for node 69e450423551964617339c97 2026-04-21 03:47:20,127 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:47:20,142 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e450423551964617339c97 HTTP/1.1" 200 None 2026-04-21 03:47:20,143 - lava_callback - DEBUG - Processing node 69e450423551964617339c97: baseline-arm-clabbe 2026-04-21 03:47:20,143 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e450423551964617339c97 2026-04-21 03:47:20,144 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:47:20,159 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e450423551964617339c97 HTTP/1.1" 200 None 2026-04-21 03:47:20,161 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:47:20,174 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:47:20,175 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:47:20,210 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e450423551964617339c97 HTTP/1.1" 200 None 2026-04-21 03:47:20,211 - lava_callback - ERROR - Error processing callback for node 69e450423551964617339c97: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54166 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54176 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58756 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42726 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54652 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50428 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35844 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35852 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:48:20,219 - lava_callback - INFO - Received callback for node 69e450753551964617339ca0 INFO: 172.20.0.1:55268 - "POST /node/69e450753551964617339ca0 HTTP/1.0" 202 Accepted 2026-04-21 03:48:20,226 - lava_callback - INFO - Starting async job submit for node 69e450753551964617339ca0 2026-04-21 03:48:20,228 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:48:20,244 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e450753551964617339ca0 HTTP/1.1" 200 None 2026-04-21 03:48:20,245 - lava_callback - DEBUG - Processing node 69e450753551964617339ca0: baseline-arm64-clabbe 2026-04-21 03:48:20,245 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e450753551964617339ca0 2026-04-21 03:48:20,246 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:48:20,265 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e450753551964617339ca0 HTTP/1.1" 200 None 2026-04-21 03:48:20,268 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:48:20,285 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 03:48:20,287 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:48:20,306 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e450753551964617339ca0 HTTP/1.1" 200 None 2026-04-21 03:48:20,307 - lava_callback - ERROR - Error processing callback for node 69e450753551964617339ca0: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:55270 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55276 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60282 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51214 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38862 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46026 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60496 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60506 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41024 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41034 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:49:35,041 - lava_callback - INFO - Received callback for node 69e450ba3551964617339cc1 INFO: 172.20.0.1:57774 - "POST /node/69e450ba3551964617339cc1 HTTP/1.0" 202 Accepted 2026-04-21 03:49:35,046 - lava_callback - INFO - Starting async job submit for node 69e450ba3551964617339cc1 2026-04-21 03:49:35,047 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:49:35,061 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e450ba3551964617339cc1 HTTP/1.1" 200 None 2026-04-21 03:49:35,062 - lava_callback - DEBUG - Processing node 69e450ba3551964617339cc1: baseline-arm64-baylibre 2026-04-21 03:49:35,062 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e450ba3551964617339cc1 2026-04-21 03:49:35,063 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:49:35,077 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e450ba3551964617339cc1 HTTP/1.1" 200 None 2026-04-21 03:49:35,079 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:49:35,091 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e33e363745d1193f227399 HTTP/1.1" 200 None 2026-04-21 03:49:35,093 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:49:35,110 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e450ba3551964617339cc1 HTTP/1.1" 200 None 2026-04-21 03:49:35,111 - lava_callback - ERROR - Error processing callback for node 69e450ba3551964617339cc1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:57780 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45106 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42978 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60408 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35904 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35916 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58678 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58680 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44470 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:50:40,280 - lava_callback - INFO - Received callback for node 69e450fd3551964617339cdc INFO: 172.20.0.1:40492 - "POST /node/69e450fd3551964617339cdc HTTP/1.0" 202 Accepted 2026-04-21 03:50:40,285 - lava_callback - INFO - Starting async job submit for node 69e450fd3551964617339cdc 2026-04-21 03:50:40,286 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:50:40,300 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e450fd3551964617339cdc HTTP/1.1" 200 None 2026-04-21 03:50:40,301 - lava_callback - DEBUG - Processing node 69e450fd3551964617339cdc: baseline-arm-clabbe 2026-04-21 03:50:40,301 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e450fd3551964617339cdc 2026-04-21 03:50:40,302 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:50:40,314 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e450fd3551964617339cdc HTTP/1.1" 200 None 2026-04-21 03:50:40,316 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:50:40,328 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:50:40,330 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:50:40,351 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e450fd3551964617339cdc HTTP/1.1" 200 None 2026-04-21 03:50:40,352 - lava_callback - ERROR - Error processing callback for node 69e450fd3551964617339cdc: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:40506 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38584 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46884 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50540 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50550 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:51:20,235 - lava_callback - INFO - Received callback for node 69e451303551964617339cf0 INFO: 172.20.0.1:34174 - "POST /node/69e451303551964617339cf0 HTTP/1.0" 202 Accepted 2026-04-21 03:51:20,244 - lava_callback - INFO - Starting async job submit for node 69e451303551964617339cf0 2026-04-21 03:51:20,245 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:51:20,259 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e451303551964617339cf0 HTTP/1.1" 200 None 2026-04-21 03:51:20,260 - lava_callback - DEBUG - Processing node 69e451303551964617339cf0: baseline-arm64-clabbe 2026-04-21 03:51:20,260 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e451303551964617339cf0 2026-04-21 03:51:20,261 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:51:20,275 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e451303551964617339cf0 HTTP/1.1" 200 None 2026-04-21 03:51:20,277 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:51:20,296 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 03:51:20,297 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:51:20,315 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e451303551964617339cf0 HTTP/1.1" 200 None 2026-04-21 03:51:20,316 - lava_callback - ERROR - Error processing callback for node 69e451303551964617339cf0: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:34182 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34188 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48440 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57888 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36036 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37922 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39700 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39708 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58504 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58510 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40958 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:52:40,362 - lava_callback - INFO - Received callback for node 69e4517b3551964617339d07 INFO: 172.20.0.1:54768 - "POST /node/69e4517b3551964617339d07 HTTP/1.0" 202 Accepted 2026-04-21 03:52:40,368 - lava_callback - INFO - Starting async job submit for node 69e4517b3551964617339d07 2026-04-21 03:52:40,368 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:52:40,383 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4517b3551964617339d07 HTTP/1.1" 200 None 2026-04-21 03:52:40,384 - lava_callback - DEBUG - Processing node 69e4517b3551964617339d07: baseline-arm64-clabbe 2026-04-21 03:52:40,384 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4517b3551964617339d07 2026-04-21 03:52:40,385 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:52:40,398 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4517b3551964617339d07 HTTP/1.1" 200 None 2026-04-21 03:52:40,399 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:52:40,412 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e33e363745d1193f227399 HTTP/1.1" 200 None 2026-04-21 03:52:40,413 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:52:40,431 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4517b3551964617339d07 HTTP/1.1" 200 None 2026-04-21 03:52:40,433 - lava_callback - ERROR - Error processing callback for node 69e4517b3551964617339d07: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54774 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45298 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48158 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43646 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43658 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51404 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51410 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51470 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:53:40,448 - lava_callback - INFO - Received callback for node 69e451b83551964617339d22 INFO: 172.20.0.1:35240 - "POST /node/69e451b83551964617339d22 HTTP/1.0" 202 Accepted 2026-04-21 03:53:40,454 - lava_callback - INFO - Starting async job submit for node 69e451b83551964617339d22 2026-04-21 03:53:40,455 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:53:40,469 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e451b83551964617339d22 HTTP/1.1" 200 None 2026-04-21 03:53:40,470 - lava_callback - DEBUG - Processing node 69e451b83551964617339d22: baseline-arm-clabbe 2026-04-21 03:53:40,470 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e451b83551964617339d22 2026-04-21 03:53:40,471 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:53:40,483 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e451b83551964617339d22 HTTP/1.1" 200 None 2026-04-21 03:53:40,484 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:53:40,508 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 03:53:40,509 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:53:40,529 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e451b83551964617339d22 HTTP/1.1" 200 None 2026-04-21 03:53:40,530 - lava_callback - ERROR - Error processing callback for node 69e451b83551964617339d22: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:35244 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42268 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60234 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44248 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44264 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41850 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41866 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58604 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:54:40,444 - lava_callback - INFO - Received callback for node 69e451eb3551964617339d33 INFO: 172.20.0.1:51390 - "POST /node/69e451eb3551964617339d33 HTTP/1.0" 202 Accepted 2026-04-21 03:54:40,450 - lava_callback - INFO - Starting async job submit for node 69e451eb3551964617339d33 2026-04-21 03:54:40,451 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:54:40,465 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e451eb3551964617339d33 HTTP/1.1" 200 None 2026-04-21 03:54:40,466 - lava_callback - DEBUG - Processing node 69e451eb3551964617339d33: baseline-arm64-clabbe 2026-04-21 03:54:40,466 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e451eb3551964617339d33 2026-04-21 03:54:40,466 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:54:40,479 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e451eb3551964617339d33 HTTP/1.1" 200 None 2026-04-21 03:54:40,481 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:54:40,493 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e33e363745d1193f227399 HTTP/1.1" 200 None 2026-04-21 03:54:40,495 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:54:40,513 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e451eb3551964617339d33 HTTP/1.1" 200 None 2026-04-21 03:54:40,514 - lava_callback - ERROR - Error processing callback for node 69e451eb3551964617339d33: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:51402 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40868 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48196 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45338 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45352 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35440 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35454 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44802 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:55:40,496 - lava_callback - INFO - Received callback for node 69e452363551964617339d4c INFO: 172.20.0.1:51338 - "POST /node/69e452363551964617339d4c HTTP/1.0" 202 Accepted 2026-04-21 03:55:40,501 - lava_callback - INFO - Starting async job submit for node 69e452363551964617339d4c 2026-04-21 03:55:40,502 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:55:40,515 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e452363551964617339d4c HTTP/1.1" 200 None 2026-04-21 03:55:40,516 - lava_callback - DEBUG - Processing node 69e452363551964617339d4c: baseline-arm64-clabbe 2026-04-21 03:55:40,516 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e452363551964617339d4c 2026-04-21 03:55:40,517 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:55:40,530 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e452363551964617339d4c HTTP/1.1" 200 None 2026-04-21 03:55:40,532 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:55:40,544 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 03:55:40,546 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:55:40,572 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e452363551964617339d4c HTTP/1.1" 200 None 2026-04-21 03:55:40,573 - lava_callback - ERROR - Error processing callback for node 69e452363551964617339d4c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:51340 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45278 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49882 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37206 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37216 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41268 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41284 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43216 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35976 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44556 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:57:00,554 - lava_callback - INFO - Received callback for node 69e452803551964617339d6f INFO: 172.20.0.1:60792 - "POST /node/69e452803551964617339d6f HTTP/1.0" 202 Accepted 2026-04-21 03:57:00,560 - lava_callback - INFO - Starting async job submit for node 69e452803551964617339d6f 2026-04-21 03:57:00,561 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:57:00,575 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e452803551964617339d6f HTTP/1.1" 200 None 2026-04-21 03:57:00,576 - lava_callback - DEBUG - Processing node 69e452803551964617339d6f: baseline-arm64-clabbe 2026-04-21 03:57:00,576 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e452803551964617339d6f 2026-04-21 03:57:00,577 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:57:00,590 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e452803551964617339d6f HTTP/1.1" 200 None 2026-04-21 03:57:00,592 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:57:00,610 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e33e363745d1193f227399 HTTP/1.1" 200 None 2026-04-21 03:57:00,612 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:57:00,630 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e452803551964617339d6f HTTP/1.1" 200 None 2026-04-21 03:57:00,631 - lava_callback - ERROR - Error processing callback for node 69e452803551964617339d6f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:60794 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48758 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48768 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47004 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47014 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46280 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53792 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51492 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41472 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52246 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52250 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:58:20,606 - lava_callback - INFO - Received callback for node 69e452ca3551964617339d88 INFO: 172.20.0.1:55124 - "POST /node/69e452ca3551964617339d88 HTTP/1.0" 202 Accepted 2026-04-21 03:58:20,611 - lava_callback - INFO - Starting async job submit for node 69e452ca3551964617339d88 2026-04-21 03:58:20,612 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:58:20,627 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e452ca3551964617339d88 HTTP/1.1" 200 None 2026-04-21 03:58:20,627 - lava_callback - DEBUG - Processing node 69e452ca3551964617339d88: baseline-arm64-clabbe 2026-04-21 03:58:20,628 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e452ca3551964617339d88 2026-04-21 03:58:20,628 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:58:20,645 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e452ca3551964617339d88 HTTP/1.1" 200 None 2026-04-21 03:58:20,646 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:58:20,659 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de9610d105963562bb3a33 HTTP/1.1" 200 None 2026-04-21 03:58:20,661 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:58:20,678 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e452ca3551964617339d88 HTTP/1.1" 200 None 2026-04-21 03:58:20,679 - lava_callback - ERROR - Error processing callback for node 69e452ca3551964617339d88: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:55138 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55140 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41272 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44216 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46526 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37624 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52672 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52684 - "GET / HTTP/1.0" 200 OK 2026-04-21 03:59:20,657 - lava_callback - INFO - Received callback for node 69e453143551964617339dad INFO: 172.20.0.1:58014 - "POST /node/69e453143551964617339dad HTTP/1.0" 202 Accepted 2026-04-21 03:59:20,663 - lava_callback - INFO - Starting async job submit for node 69e453143551964617339dad 2026-04-21 03:59:20,664 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:59:20,683 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e453143551964617339dad HTTP/1.1" 200 None 2026-04-21 03:59:20,684 - lava_callback - DEBUG - Processing node 69e453143551964617339dad: baseline-arm64-clabbe 2026-04-21 03:59:20,684 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e453143551964617339dad 2026-04-21 03:59:20,685 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:59:20,704 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e453143551964617339dad HTTP/1.1" 200 None 2026-04-21 03:59:20,707 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:59:20,727 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e33e363745d1193f227399 HTTP/1.1" 200 None 2026-04-21 03:59:20,729 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 03:59:20,747 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e453143551964617339dad HTTP/1.1" 200 None 2026-04-21 03:59:20,748 - lava_callback - ERROR - Error processing callback for node 69e453143551964617339dad: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:58016 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58020 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52994 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56448 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51778 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53320 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54736 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:54748 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35420 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35434 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57400 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:00:40,763 - lava_callback - INFO - Received callback for node 69e453613551964617339dc5 INFO: 172.20.0.1:44834 - "POST /node/69e453613551964617339dc5 HTTP/1.0" 202 Accepted 2026-04-21 04:00:40,769 - lava_callback - INFO - Starting async job submit for node 69e453613551964617339dc5 2026-04-21 04:00:40,770 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:00:40,784 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e453613551964617339dc5 HTTP/1.1" 200 None 2026-04-21 04:00:40,785 - lava_callback - DEBUG - Processing node 69e453613551964617339dc5: baseline-arm64-clabbe 2026-04-21 04:00:40,785 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e453613551964617339dc5 2026-04-21 04:00:40,786 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:00:40,798 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e453613551964617339dc5 HTTP/1.1" 200 None 2026-04-21 04:00:40,800 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:00:40,815 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e33e363745d1193f227399 HTTP/1.1" 200 None 2026-04-21 04:00:40,817 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:00:40,834 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e453613551964617339dc5 HTTP/1.1" 200 None 2026-04-21 04:00:40,835 - lava_callback - ERROR - Error processing callback for node 69e453613551964617339dc5: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:44842 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39808 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58112 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54634 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:54650 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40314 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40324 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35106 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42486 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56392 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:02:00,800 - lava_callback - INFO - Received callback for node 69e453ad3551964617339de6 INFO: 172.20.0.1:42572 - "POST /node/69e453ad3551964617339de6 HTTP/1.0" 202 Accepted 2026-04-21 04:02:00,805 - lava_callback - INFO - Starting async job submit for node 69e453ad3551964617339de6 2026-04-21 04:02:00,806 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:02:00,829 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e453ad3551964617339de6 HTTP/1.1" 200 None 2026-04-21 04:02:00,830 - lava_callback - DEBUG - Processing node 69e453ad3551964617339de6: baseline-arm64-clabbe 2026-04-21 04:02:00,830 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e453ad3551964617339de6 2026-04-21 04:02:00,831 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:02:00,843 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e453ad3551964617339de6 HTTP/1.1" 200 None 2026-04-21 04:02:00,845 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:02:00,868 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e33e363745d1193f227399 HTTP/1.1" 200 None 2026-04-21 04:02:00,869 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:02:00,886 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e453ad3551964617339de6 HTTP/1.1" 200 None 2026-04-21 04:02:00,887 - lava_callback - ERROR - Error processing callback for node 69e453ad3551964617339de6: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:42586 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35390 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35406 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55766 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55780 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58908 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59780 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50478 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58048 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54252 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:54266 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:03:20,848 - lava_callback - INFO - Received callback for node 69e453f83551964617339e00 INFO: 172.20.0.1:55664 - "POST /node/69e453f83551964617339e00 HTTP/1.0" 202 Accepted 2026-04-21 04:03:20,854 - lava_callback - INFO - Starting async job submit for node 69e453f83551964617339e00 2026-04-21 04:03:20,855 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:03:20,870 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e453f83551964617339e00 HTTP/1.1" 200 None 2026-04-21 04:03:20,870 - lava_callback - DEBUG - Processing node 69e453f83551964617339e00: baseline-arm64-clabbe 2026-04-21 04:03:20,871 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e453f83551964617339e00 2026-04-21 04:03:20,871 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:03:20,884 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e453f83551964617339e00 HTTP/1.1" 200 None 2026-04-21 04:03:20,886 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:03:20,909 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e33e363745d1193f227399 HTTP/1.1" 200 None 2026-04-21 04:03:20,910 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:03:20,928 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e453f83551964617339e00 HTTP/1.1" 200 None 2026-04-21 04:03:20,929 - lava_callback - ERROR - Error processing callback for node 69e453f83551964617339e00: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:55678 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55680 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58918 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51220 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35026 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41218 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46188 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46194 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44270 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44280 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41812 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45750 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36792 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43444 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37144 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37158 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59706 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59714 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40080 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57692 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55684 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33966 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50294 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50298 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57456 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57468 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48950 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57916 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38788 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:07:01,118 - lava_callback - INFO - Received callback for node 69e6ef4760f5826562c16ed4 INFO: 172.20.0.1:36332 - "POST /node/69e6ef4760f5826562c16ed4 HTTP/1.0" 202 Accepted 2026-04-21 04:07:01,127 - lava_callback - INFO - Starting async job submit for node 69e6ef4760f5826562c16ed4 2026-04-21 04:07:01,170 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:07:01,196 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6ef4760f5826562c16ed4 HTTP/1.1" 200 None 2026-04-21 04:07:01,197 - lava_callback - DEBUG - Processing node 69e6ef4760f5826562c16ed4: baseline-nfs-arm64 2026-04-21 04:07:02,388 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 04:07:02,546 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 04:07:02,548 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6ef4760f5826562c16ed4/log.txt.gz 2026-04-21 04:07:03,206 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 04:07:03,441 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 04:07:03,443 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6ef4760f5826562c16ed4/lava-logs.yaml 2026-04-21 04:07:03,465 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 04:07:03,598 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 04:07:03,600 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6ef4760f5826562c16ed4/lava_callback.json.gz DEBUG: Setup failed for 69e6ef4760f5826562c16ed4. Transitting job node result: incomplete -> fail 2026-04-21 04:07:03,687 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:07:03,701 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6ef4760f5826562c16ed4 HTTP/1.1" 200 None 2026-04-21 04:07:03,703 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:07:03,716 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 04:07:03,717 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:07:03,736 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6ef4760f5826562c16ed4 HTTP/1.1" 200 None 2026-04-21 04:07:03,825 - lava_callback - INFO - Completed processing callback for node 69e6ef4760f5826562c16ed4 INFO: 172.20.0.1:36344 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42822 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 04:07:15,944 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:07:15,974 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:42834 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39722 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39732 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47434 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36842 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36162 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32826 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48342 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48344 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47460 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47466 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37442 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60990 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48034 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46020 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57618 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57624 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55648 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55656 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38172 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46364 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37646 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59436 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43554 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43566 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55904 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55906 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54410 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53438 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33338 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60960 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51438 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51440 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42094 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42098 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54454 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52554 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38226 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33030 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34044 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34052 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58844 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58852 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40060 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42134 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48330 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47788 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37406 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37414 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36086 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36096 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38054 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34576 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49128 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33530 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38602 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:38606 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41978 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41986 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45872 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54046 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40460 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44680 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44294 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44300 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53090 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53102 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48796 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39540 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37142 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50980 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52584 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52596 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42330 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42336 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56012 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56660 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38476 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33160 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37218 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37226 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60062 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60066 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48494 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36574 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51986 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43534 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46044 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46048 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38232 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38248 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46368 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38172 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42840 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34422 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52994 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53000 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45880 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45896 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32818 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44120 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54052 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42198 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46658 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46660 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38248 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38254 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47152 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38830 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49666 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58492 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50112 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50124 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46874 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46880 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58378 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43656 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40464 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48034 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36096 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:36110 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46106 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46114 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47352 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46042 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50814 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43646 - "GET /node/69840341969566dcb0d2b7fb%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:43650 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58968 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58980 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37974 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37988 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34704 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51204 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35668 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59554 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41586 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41590 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52954 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52964 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55650 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55522 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48156 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56962 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46604 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46608 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41210 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41218 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42814 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57314 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39362 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52004 - "GET /node/6986fc156e077e4126a7d4c8%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:52006 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33978 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33986 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55028 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55034 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60358 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53486 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43186 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44534 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57296 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57310 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46986 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46998 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39702 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40566 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59970 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49286 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37554 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37566 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40778 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40792 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60500 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52306 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59804 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37162 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48206 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48210 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35418 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35432 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34134 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:29:37,280 - lava_callback - INFO - Received callback for node 69e45a243551964617339e27 INFO: 172.20.0.1:34150 - "POST /node/69e45a243551964617339e27 HTTP/1.0" 202 Accepted 2026-04-21 04:29:37,285 - lava_callback - INFO - Starting async job submit for node 69e45a243551964617339e27 2026-04-21 04:29:37,286 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:29:37,310 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45a243551964617339e27 HTTP/1.1" 200 None 2026-04-21 04:29:37,311 - lava_callback - DEBUG - Processing node 69e45a243551964617339e27: baseline-arm-baylibre 2026-04-21 04:29:37,311 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e45a243551964617339e27 2026-04-21 04:29:37,312 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:29:37,324 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45a243551964617339e27 HTTP/1.1" 200 None 2026-04-21 04:29:37,325 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:29:37,338 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 04:29:37,340 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:29:37,356 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e45a243551964617339e27 HTTP/1.1" 200 None 2026-04-21 04:29:37,357 - lava_callback - ERROR - Error processing callback for node 69e45a243551964617339e27: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:59200 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37776 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48404 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51146 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51158 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:30:17,152 - lava_callback - INFO - Received callback for node 69e45a4c3551964617339e34 INFO: 172.20.0.1:51168 - "POST /node/69e45a4c3551964617339e34 HTTP/1.0" 202 Accepted 2026-04-21 04:30:17,158 - lava_callback - INFO - Starting async job submit for node 69e45a4c3551964617339e34 2026-04-21 04:30:17,159 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:30:17,191 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45a4c3551964617339e34 HTTP/1.1" 200 None 2026-04-21 04:30:17,192 - lava_callback - DEBUG - Processing node 69e45a4c3551964617339e34: baseline-arm-baylibre 2026-04-21 04:30:17,192 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e45a4c3551964617339e34 2026-04-21 04:30:17,193 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:30:17,214 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45a4c3551964617339e34 HTTP/1.1" 200 None 2026-04-21 04:30:17,216 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:30:17,229 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 04:30:17,230 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:30:17,252 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e45a4c3551964617339e34 HTTP/1.1" 200 None 2026-04-21 04:30:17,253 - lava_callback - ERROR - Error processing callback for node 69e45a4c3551964617339e34: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47162 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47166 - "GET /node/69862f65177ca0f097bd2415%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:47178 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34664 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53172 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37612 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:30:57,374 - lava_callback - INFO - Received callback for node 69e45a753551964617339e41 INFO: 172.20.0.1:37616 - "POST /node/69e45a753551964617339e41 HTTP/1.0" 202 Accepted 2026-04-21 04:30:57,380 - lava_callback - INFO - Starting async job submit for node 69e45a753551964617339e41 2026-04-21 04:30:57,381 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:30:57,400 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45a753551964617339e41 HTTP/1.1" 200 None 2026-04-21 04:30:57,401 - lava_callback - DEBUG - Processing node 69e45a753551964617339e41: baseline-arm-baylibre 2026-04-21 04:30:57,401 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e45a753551964617339e41 2026-04-21 04:30:57,402 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:30:57,415 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45a753551964617339e41 HTTP/1.1" 200 None 2026-04-21 04:30:57,416 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:30:57,429 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 04:30:57,430 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:30:57,447 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e45a753551964617339e41 HTTP/1.1" 200 None 2026-04-21 04:30:57,448 - lava_callback - ERROR - Error processing callback for node 69e45a753551964617339e41: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:40192 - "GET /node/6985c4d371989fb1414980b6%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:40202 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50636 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50650 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56314 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56324 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53534 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:31:37,218 - lava_callback - INFO - Received callback for node 69e45a9d3551964617339e4e INFO: 172.20.0.1:53538 - "POST /node/69e45a9d3551964617339e4e HTTP/1.0" 202 Accepted 2026-04-21 04:31:37,223 - lava_callback - INFO - Starting async job submit for node 69e45a9d3551964617339e4e 2026-04-21 04:31:37,224 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:31:37,238 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45a9d3551964617339e4e HTTP/1.1" 200 None 2026-04-21 04:31:37,239 - lava_callback - DEBUG - Processing node 69e45a9d3551964617339e4e: baseline-arm-baylibre 2026-04-21 04:31:37,240 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e45a9d3551964617339e4e 2026-04-21 04:31:37,240 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:31:37,256 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45a9d3551964617339e4e HTTP/1.1" 200 None 2026-04-21 04:31:37,258 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:31:37,271 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 04:31:37,273 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:31:37,290 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e45a9d3551964617339e4e HTTP/1.1" 200 None 2026-04-21 04:31:37,291 - lava_callback - ERROR - Error processing callback for node 69e45a9d3551964617339e4e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:57600 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35872 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47794 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39874 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39882 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:32:17,291 - lava_callback - INFO - Received callback for node 69e45ac73551964617339e5b INFO: 172.20.0.1:39888 - "POST /node/69e45ac73551964617339e5b HTTP/1.0" 202 Accepted 2026-04-21 04:32:17,297 - lava_callback - INFO - Starting async job submit for node 69e45ac73551964617339e5b 2026-04-21 04:32:17,298 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:32:17,333 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45ac73551964617339e5b HTTP/1.1" 200 None 2026-04-21 04:32:17,334 - lava_callback - DEBUG - Processing node 69e45ac73551964617339e5b: baseline-arm-baylibre 2026-04-21 04:32:17,334 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e45ac73551964617339e5b 2026-04-21 04:32:17,335 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:32:17,349 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45ac73551964617339e5b HTTP/1.1" 200 None 2026-04-21 04:32:17,350 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:32:17,369 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 04:32:17,371 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:32:17,389 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e45ac73551964617339e5b HTTP/1.1" 200 None 2026-04-21 04:32:17,390 - lava_callback - ERROR - Error processing callback for node 69e45ac73551964617339e5b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49700 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49712 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47480 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57830 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39596 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:32:57,305 - lava_callback - INFO - Received callback for node 69e45aed3551964617339e6a INFO: 172.20.0.1:39602 - "POST /node/69e45aed3551964617339e6a HTTP/1.0" 202 Accepted 2026-04-21 04:32:57,311 - lava_callback - INFO - Starting async job submit for node 69e45aed3551964617339e6a 2026-04-21 04:32:57,312 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:32:57,330 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45aed3551964617339e6a HTTP/1.1" 200 None 2026-04-21 04:32:57,331 - lava_callback - DEBUG - Processing node 69e45aed3551964617339e6a: baseline-arm-baylibre 2026-04-21 04:32:57,331 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e45aed3551964617339e6a 2026-04-21 04:32:57,332 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:32:57,345 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45aed3551964617339e6a HTTP/1.1" 200 None 2026-04-21 04:32:57,346 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:32:57,360 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 04:32:57,361 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:32:57,385 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e45aed3551964617339e6a HTTP/1.1" 200 None 2026-04-21 04:32:57,386 - lava_callback - ERROR - Error processing callback for node 69e45aed3551964617339e6a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:35828 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51060 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51072 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49510 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49514 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41444 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50858 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58774 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:34:02,167 - lava_callback - INFO - Received callback for node 69e45b2b3551964617339e77 INFO: 172.20.0.1:34570 - "POST /node/69e45b2b3551964617339e77 HTTP/1.0" 202 Accepted 2026-04-21 04:34:02,172 - lava_callback - INFO - Starting async job submit for node 69e45b2b3551964617339e77 2026-04-21 04:34:02,173 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:34:02,188 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45b2b3551964617339e77 HTTP/1.1" 200 None 2026-04-21 04:34:02,188 - lava_callback - DEBUG - Processing node 69e45b2b3551964617339e77: baseline-arm-clabbe 2026-04-21 04:34:02,189 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e45b2b3551964617339e77 2026-04-21 04:34:02,189 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:34:02,202 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45b2b3551964617339e77 HTTP/1.1" 200 None 2026-04-21 04:34:02,204 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:34:02,216 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 04:34:02,218 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:34:02,243 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e45b2b3551964617339e77 HTTP/1.1" 200 None 2026-04-21 04:34:02,244 - lava_callback - ERROR - Error processing callback for node 69e45b2b3551964617339e77: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:34580 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41152 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41166 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32812 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32814 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51256 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:34:42,146 - lava_callback - INFO - Received callback for node 69e45b533551964617339e82 INFO: 172.20.0.1:40324 - "POST /node/69e45b533551964617339e82 HTTP/1.0" 202 Accepted 2026-04-21 04:34:42,152 - lava_callback - INFO - Starting async job submit for node 69e45b533551964617339e82 2026-04-21 04:34:42,153 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:34:42,171 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45b533551964617339e82 HTTP/1.1" 200 None 2026-04-21 04:34:42,172 - lava_callback - DEBUG - Processing node 69e45b533551964617339e82: baseline-arm-clabbe 2026-04-21 04:34:42,172 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e45b533551964617339e82 2026-04-21 04:34:42,173 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:34:42,186 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45b533551964617339e82 HTTP/1.1" 200 None 2026-04-21 04:34:42,188 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:34:42,205 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 04:34:42,207 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:34:42,232 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e45b533551964617339e82 HTTP/1.1" 200 None 2026-04-21 04:34:42,233 - lava_callback - ERROR - Error processing callback for node 69e45b533551964617339e82: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:40334 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58814 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52848 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35004 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35018 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:35:22,230 - lava_callback - INFO - Received callback for node 69e45b7b3551964617339e8f INFO: 172.20.0.1:35772 - "POST /node/69e45b7b3551964617339e8f HTTP/1.0" 202 Accepted 2026-04-21 04:35:22,236 - lava_callback - INFO - Starting async job submit for node 69e45b7b3551964617339e8f 2026-04-21 04:35:22,237 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:35:22,251 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45b7b3551964617339e8f HTTP/1.1" 200 None 2026-04-21 04:35:22,252 - lava_callback - DEBUG - Processing node 69e45b7b3551964617339e8f: baseline-arm-clabbe 2026-04-21 04:35:22,252 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e45b7b3551964617339e8f 2026-04-21 04:35:22,253 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:35:22,265 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45b7b3551964617339e8f HTTP/1.1" 200 None 2026-04-21 04:35:22,267 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:35:22,288 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 04:35:22,290 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:35:22,307 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e45b7b3551964617339e8f HTTP/1.1" 200 None 2026-04-21 04:35:22,308 - lava_callback - ERROR - Error processing callback for node 69e45b7b3551964617339e8f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:35788 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35792 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56558 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54850 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57012 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:36:02,203 - lava_callback - INFO - Received callback for node 69e45ba33551964617339e9e INFO: 172.20.0.1:51126 - "POST /node/69e45ba33551964617339e9e HTTP/1.0" 202 Accepted 2026-04-21 04:36:02,209 - lava_callback - INFO - Starting async job submit for node 69e45ba33551964617339e9e 2026-04-21 04:36:02,210 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:36:02,224 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45ba33551964617339e9e HTTP/1.1" 200 None 2026-04-21 04:36:02,225 - lava_callback - DEBUG - Processing node 69e45ba33551964617339e9e: baseline-arm-clabbe 2026-04-21 04:36:02,225 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e45ba33551964617339e9e 2026-04-21 04:36:02,226 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:36:02,239 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45ba33551964617339e9e HTTP/1.1" 200 None 2026-04-21 04:36:02,240 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:36:02,254 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 04:36:02,256 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:36:02,275 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e45ba33551964617339e9e HTTP/1.1" 200 None 2026-04-21 04:36:02,276 - lava_callback - ERROR - Error processing callback for node 69e45ba33551964617339e9e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:51138 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53606 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53612 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36624 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36634 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60914 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:36:42,254 - lava_callback - INFO - Received callback for node 69e45bcb3551964617339ea9 INFO: 172.20.0.1:38516 - "POST /node/69e45bcb3551964617339ea9 HTTP/1.0" 202 Accepted 2026-04-21 04:36:42,260 - lava_callback - INFO - Starting async job submit for node 69e45bcb3551964617339ea9 2026-04-21 04:36:42,261 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:36:42,275 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45bcb3551964617339ea9 HTTP/1.1" 200 None 2026-04-21 04:36:42,276 - lava_callback - DEBUG - Processing node 69e45bcb3551964617339ea9: baseline-arm-clabbe 2026-04-21 04:36:42,276 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e45bcb3551964617339ea9 2026-04-21 04:36:42,277 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:36:42,289 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45bcb3551964617339ea9 HTTP/1.1" 200 None 2026-04-21 04:36:42,291 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:36:42,304 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 04:36:42,305 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:36:42,323 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e45bcb3551964617339ea9 HTTP/1.1" 200 None 2026-04-21 04:36:42,324 - lava_callback - ERROR - Error processing callback for node 69e45bcb3551964617339ea9: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:38520 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40722 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35758 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45292 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45300 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:37:22,262 - lava_callback - INFO - Received callback for node 69e45bf33551964617339eb6 INFO: 172.20.0.1:54710 - "POST /node/69e45bf33551964617339eb6 HTTP/1.0" 202 Accepted 2026-04-21 04:37:22,268 - lava_callback - INFO - Starting async job submit for node 69e45bf33551964617339eb6 2026-04-21 04:37:22,269 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:37:22,292 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45bf33551964617339eb6 HTTP/1.1" 200 None 2026-04-21 04:37:22,293 - lava_callback - DEBUG - Processing node 69e45bf33551964617339eb6: baseline-arm-clabbe 2026-04-21 04:37:22,293 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e45bf33551964617339eb6 2026-04-21 04:37:22,294 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:37:22,306 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45bf33551964617339eb6 HTTP/1.1" 200 None 2026-04-21 04:37:22,308 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:37:22,321 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 04:37:22,322 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:37:22,352 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e45bf33551964617339eb6 HTTP/1.1" 200 None 2026-04-21 04:37:22,354 - lava_callback - ERROR - Error processing callback for node 69e45bf33551964617339eb6: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54720 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54728 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51544 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34870 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57188 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:38:02,335 - lava_callback - INFO - Received callback for node 69e45c183551964617339ec6 INFO: 172.20.0.1:36584 - "POST /node/69e45c183551964617339ec6 HTTP/1.0" 202 Accepted 2026-04-21 04:38:02,340 - lava_callback - INFO - Starting async job submit for node 69e45c183551964617339ec6 2026-04-21 04:38:02,341 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:38:02,356 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45c183551964617339ec6 HTTP/1.1" 200 None 2026-04-21 04:38:02,356 - lava_callback - DEBUG - Processing node 69e45c183551964617339ec6: baseline-arm-clabbe 2026-04-21 04:38:02,357 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e45c183551964617339ec6 2026-04-21 04:38:02,357 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:38:02,370 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45c183551964617339ec6 HTTP/1.1" 200 None 2026-04-21 04:38:02,372 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:38:02,388 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 04:38:02,390 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:38:02,418 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e45c183551964617339ec6 HTTP/1.1" 200 None 2026-04-21 04:38:02,419 - lava_callback - ERROR - Error processing callback for node 69e45c183551964617339ec6: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:36590 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41404 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41412 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46902 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46914 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55600 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:38:42,304 - lava_callback - INFO - Received callback for node 69e45c3d3551964617339ed1 INFO: 172.20.0.1:51984 - "POST /node/69e45c3d3551964617339ed1 HTTP/1.0" 202 Accepted 2026-04-21 04:38:42,310 - lava_callback - INFO - Starting async job submit for node 69e45c3d3551964617339ed1 2026-04-21 04:38:42,311 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:38:42,325 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45c3d3551964617339ed1 HTTP/1.1" 200 None 2026-04-21 04:38:42,326 - lava_callback - DEBUG - Processing node 69e45c3d3551964617339ed1: baseline-arm-clabbe 2026-04-21 04:38:42,326 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e45c3d3551964617339ed1 2026-04-21 04:38:42,327 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:38:42,340 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45c3d3551964617339ed1 HTTP/1.1" 200 None 2026-04-21 04:38:42,342 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:38:42,355 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 04:38:42,357 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:38:42,374 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e45c3d3551964617339ed1 HTTP/1.1" 200 None 2026-04-21 04:38:42,375 - lava_callback - ERROR - Error processing callback for node 69e45c3d3551964617339ed1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:52000 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51994 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:39:02,353 - lava_callback - INFO - Received callback for node 69e45c633551964617339ede INFO: 172.20.0.1:33294 - "POST /node/69e45c633551964617339ede HTTP/1.0" 202 Accepted 2026-04-21 04:39:02,359 - lava_callback - INFO - Starting async job submit for node 69e45c633551964617339ede 2026-04-21 04:39:02,360 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:39:02,374 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45c633551964617339ede HTTP/1.1" 200 None 2026-04-21 04:39:02,375 - lava_callback - DEBUG - Processing node 69e45c633551964617339ede: baseline-arm-clabbe 2026-04-21 04:39:02,375 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e45c633551964617339ede 2026-04-21 04:39:02,376 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:39:02,393 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45c633551964617339ede HTTP/1.1" 200 None 2026-04-21 04:39:02,395 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:39:02,407 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69de95f8d105963562bb39ff HTTP/1.1" 200 None 2026-04-21 04:39:02,408 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:39:02,426 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e45c633551964617339ede HTTP/1.1" 200 None 2026-04-21 04:39:02,427 - lava_callback - ERROR - Error processing callback for node 69e45c633551964617339ede: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:33296 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55262 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:55276 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45902 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45918 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35506 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48220 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52196 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60632 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45884 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45900 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44452 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44454 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41426 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56910 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50894 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60062 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54696 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:54698 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40112 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40116 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45824 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37868 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43024 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59124 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50956 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50960 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40854 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40864 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37884 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49980 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32946 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59034 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41492 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41506 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50300 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50316 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43328 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43890 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46186 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41898 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56792 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:56806 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52644 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52660 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34054 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46744 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47772 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34394 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52180 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52182 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:45:18,004 - lava_callback - INFO - Received callback for node 69e45dd33551964617339f07 INFO: 172.20.0.1:52190 - "POST /node/69e45dd33551964617339f07 HTTP/1.0" 202 Accepted 2026-04-21 04:45:18,010 - lava_callback - INFO - Starting async job submit for node 69e45dd33551964617339f07 2026-04-21 04:45:18,011 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:45:18,025 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45dd33551964617339f07 HTTP/1.1" 200 None 2026-04-21 04:45:18,026 - lava_callback - DEBUG - Processing node 69e45dd33551964617339f07: baseline-arm-baylibre 2026-04-21 04:45:18,026 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e45dd33551964617339f07 2026-04-21 04:45:18,027 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:45:18,039 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45dd33551964617339f07 HTTP/1.1" 200 None 2026-04-21 04:45:18,041 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:45:18,058 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e3aefd7f5347282fbdbc78 HTTP/1.1" 200 None 2026-04-21 04:45:18,059 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:45:18,076 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e45dd33551964617339f07 HTTP/1.1" 200 None 2026-04-21 04:45:18,078 - lava_callback - ERROR - Error processing callback for node 69e45dd33551964617339f07: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:51366 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51370 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57608 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45564 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52608 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:45:58,000 - lava_callback - INFO - Received callback for node 69e45e043551964617339f16 INFO: 172.20.0.1:52624 - "POST /node/69e45e043551964617339f16 HTTP/1.0" 202 Accepted 2026-04-21 04:45:58,006 - lava_callback - INFO - Starting async job submit for node 69e45e043551964617339f16 2026-04-21 04:45:58,007 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:45:58,023 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45e043551964617339f16 HTTP/1.1" 200 None 2026-04-21 04:45:58,024 - lava_callback - DEBUG - Processing node 69e45e043551964617339f16: baseline-arm-baylibre 2026-04-21 04:45:58,024 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e45e043551964617339f16 2026-04-21 04:45:58,024 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:45:58,037 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45e043551964617339f16 HTTP/1.1" 200 None 2026-04-21 04:45:58,039 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:45:58,052 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e3aefd7f5347282fbdbc78 HTTP/1.1" 200 None 2026-04-21 04:45:58,053 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:45:58,073 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e45e043551964617339f16 HTTP/1.1" 200 None 2026-04-21 04:45:58,074 - lava_callback - ERROR - Error processing callback for node 69e45e043551964617339f16: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:38318 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57378 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57386 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35134 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35138 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51858 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51016 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45142 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53332 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45024 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45040 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:47:18,243 - lava_callback - INFO - Received callback for node 69e45e443551964617339f24 INFO: 172.20.0.1:45046 - "POST /node/69e45e443551964617339f24 HTTP/1.0" 202 Accepted 2026-04-21 04:47:18,249 - lava_callback - INFO - Starting async job submit for node 69e45e443551964617339f24 2026-04-21 04:47:18,250 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:47:18,265 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45e443551964617339f24 HTTP/1.1" 200 None 2026-04-21 04:47:18,266 - lava_callback - DEBUG - Processing node 69e45e443551964617339f24: baseline-arm-baylibre 2026-04-21 04:47:18,266 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e45e443551964617339f24 2026-04-21 04:47:18,267 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:47:18,281 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45e443551964617339f24 HTTP/1.1" 200 None 2026-04-21 04:47:18,282 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:47:18,295 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e3aefd7f5347282fbdbc78 HTTP/1.1" 200 None 2026-04-21 04:47:18,297 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:47:18,314 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e45e443551964617339f24 HTTP/1.1" 200 None 2026-04-21 04:47:18,315 - lava_callback - ERROR - Error processing callback for node 69e45e443551964617339f24: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49790 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49806 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36422 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42328 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48552 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52312 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40906 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40918 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:48:18,162 - lava_callback - INFO - Received callback for node 69e45e7e3551964617339f35 INFO: 172.20.0.1:40932 - "POST /node/69e45e7e3551964617339f35 HTTP/1.0" 202 Accepted 2026-04-21 04:48:18,170 - lava_callback - INFO - Starting async job submit for node 69e45e7e3551964617339f35 2026-04-21 04:48:18,172 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:48:18,190 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45e7e3551964617339f35 HTTP/1.1" 200 None 2026-04-21 04:48:18,191 - lava_callback - DEBUG - Processing node 69e45e7e3551964617339f35: baseline-arm-baylibre 2026-04-21 04:48:18,191 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e45e7e3551964617339f35 2026-04-21 04:48:18,192 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:48:18,211 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45e7e3551964617339f35 HTTP/1.1" 200 None 2026-04-21 04:48:18,213 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:48:18,232 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e3aefd7f5347282fbdbc78 HTTP/1.1" 200 None 2026-04-21 04:48:18,233 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:48:18,259 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e45e7e3551964617339f35 HTTP/1.1" 200 None 2026-04-21 04:48:18,260 - lava_callback - ERROR - Error processing callback for node 69e45e7e3551964617339f35: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:48394 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48398 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52092 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:48:38,148 - lava_callback - INFO - Received callback for node 69e45ea23551964617339f44 INFO: 172.20.0.1:52098 - "POST /node/69e45ea23551964617339f44 HTTP/1.0" 202 Accepted 2026-04-21 04:48:38,157 - lava_callback - INFO - Starting async job submit for node 69e45ea23551964617339f44 2026-04-21 04:48:38,158 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:48:38,180 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45ea23551964617339f44 HTTP/1.1" 200 None 2026-04-21 04:48:38,181 - lava_callback - DEBUG - Processing node 69e45ea23551964617339f44: baseline-arm-baylibre 2026-04-21 04:48:38,181 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e45ea23551964617339f44 2026-04-21 04:48:38,182 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:48:38,201 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45ea23551964617339f44 HTTP/1.1" 200 None 2026-04-21 04:48:38,203 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:48:38,217 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e3aefd7f5347282fbdbc78 HTTP/1.1" 200 None 2026-04-21 04:48:38,220 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:48:38,237 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e45ea23551964617339f44 HTTP/1.1" 200 None 2026-04-21 04:48:38,238 - lava_callback - ERROR - Error processing callback for node 69e45ea23551964617339f44: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:59444 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48470 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42128 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:49:09,912 - lava_callback - INFO - Received callback for node 69e6efea60f5826562c16f15 INFO: 172.20.0.1:53100 - "POST /node/69e6efea60f5826562c16f15 HTTP/1.0" 202 Accepted 2026-04-21 04:49:09,917 - lava_callback - INFO - Starting async job submit for node 69e6efea60f5826562c16f15 2026-04-21 04:49:09,937 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:49:09,951 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6efea60f5826562c16f15 HTTP/1.1" 200 None 2026-04-21 04:49:09,952 - lava_callback - DEBUG - Processing node 69e6efea60f5826562c16f15: baseline-arm64 2026-04-21 04:49:10,196 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 04:49:10,477 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 04:49:10,479 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6efea60f5826562c16f15/log.txt.gz 2026-04-21 04:49:10,603 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 04:49:10,763 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 04:49:10,765 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6efea60f5826562c16f15/lava-logs.yaml 2026-04-21 04:49:10,772 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 04:49:10,881 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 04:49:10,882 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6efea60f5826562c16f15/lava_callback.json.gz DEBUG: Setup failed for 69e6efea60f5826562c16f15. Transitting job node result: incomplete -> fail 2026-04-21 04:49:10,922 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:49:10,945 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6efea60f5826562c16f15 HTTP/1.1" 200 None 2026-04-21 04:49:10,947 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:49:10,960 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 04:49:10,962 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:49:10,982 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6efea60f5826562c16f15 HTTP/1.1" 200 None 2026-04-21 04:49:11,041 - lava_callback - INFO - Completed processing callback for node 69e6efea60f5826562c16f15 INFO: 172.20.0.1:53114 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 04:49:15,984 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:49:16,002 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:53118 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33386 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33402 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50786 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:49:39,354 - lava_callback - INFO - Received callback for node 69e6f04260f5826562c16f39 INFO: 172.20.0.1:50798 - "POST /node/69e6f04260f5826562c16f39 HTTP/1.0" 202 Accepted 2026-04-21 04:49:39,360 - lava_callback - INFO - Starting async job submit for node 69e6f04260f5826562c16f39 2026-04-21 04:49:39,382 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:49:39,396 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6f04260f5826562c16f39 HTTP/1.1" 200 None 2026-04-21 04:49:39,397 - lava_callback - DEBUG - Processing node 69e6f04260f5826562c16f39: baseline-nfs-arm64 2026-04-21 04:49:39,601 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 04:49:39,726 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 04:49:39,728 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6f04260f5826562c16f39/log.txt.gz 2026-04-21 04:49:39,842 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 04:49:39,990 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 04:49:39,993 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6f04260f5826562c16f39/lava-logs.yaml 2026-04-21 04:49:39,999 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 04:49:40,106 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 04:49:40,108 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6f04260f5826562c16f39/lava_callback.json.gz DEBUG: Setup failed for 69e6f04260f5826562c16f39. Transitting job node result: incomplete -> fail 2026-04-21 04:49:40,150 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:49:40,169 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6f04260f5826562c16f39 HTTP/1.1" 200 None 2026-04-21 04:49:40,171 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:49:40,185 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 04:49:40,186 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:49:40,206 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6f04260f5826562c16f39 HTTP/1.1" 200 None 2026-04-21 04:49:40,259 - lava_callback - INFO - Completed processing callback for node 69e6f04260f5826562c16f39 2026-04-21 04:49:46,004 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:49:46,027 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:57722 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58608 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33292 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55174 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:55186 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32946 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32958 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56644 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43320 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43344 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56650 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58428 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58444 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:51:22,906 - lava_callback - INFO - Received callback for node 69e45f343551964617339f75 INFO: 172.20.0.1:59552 - "POST /node/69e45f343551964617339f75 HTTP/1.0" 202 Accepted 2026-04-21 04:51:22,912 - lava_callback - INFO - Starting async job submit for node 69e45f343551964617339f75 2026-04-21 04:51:22,913 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:51:22,942 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45f343551964617339f75 HTTP/1.1" 200 None 2026-04-21 04:51:22,943 - lava_callback - DEBUG - Processing node 69e45f343551964617339f75: baseline-arm-clabbe 2026-04-21 04:51:22,943 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e45f343551964617339f75 2026-04-21 04:51:22,944 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:51:22,976 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45f343551964617339f75 HTTP/1.1" 200 None 2026-04-21 04:51:22,977 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:51:22,990 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e3aefd7f5347282fbdbc78 HTTP/1.1" 200 None 2026-04-21 04:51:22,991 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:51:23,037 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e45f343551964617339f75 HTTP/1.1" 200 None 2026-04-21 04:51:23,038 - lava_callback - ERROR - Error processing callback for node 69e45f343551964617339f75: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:59556 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59564 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60298 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:51:42,924 - lava_callback - INFO - Received callback for node 69e45f593551964617339f84 INFO: 172.20.0.1:55470 - "POST /node/69e45f593551964617339f84 HTTP/1.0" 202 Accepted 2026-04-21 04:51:42,930 - lava_callback - INFO - Starting async job submit for node 69e45f593551964617339f84 2026-04-21 04:51:42,931 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:51:42,944 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45f593551964617339f84 HTTP/1.1" 200 None 2026-04-21 04:51:42,945 - lava_callback - DEBUG - Processing node 69e45f593551964617339f84: baseline-arm-clabbe 2026-04-21 04:51:42,946 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e45f593551964617339f84 2026-04-21 04:51:42,946 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:51:42,959 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45f593551964617339f84 HTTP/1.1" 200 None 2026-04-21 04:51:42,960 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:51:42,973 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e3aefd7f5347282fbdbc78 HTTP/1.1" 200 None 2026-04-21 04:51:42,975 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:51:42,992 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e45f593551964617339f84 HTTP/1.1" 200 None 2026-04-21 04:51:42,994 - lava_callback - ERROR - Error processing callback for node 69e45f593551964617339f84: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:55486 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33524 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49356 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54528 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:54538 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:52:22,950 - lava_callback - INFO - Received callback for node 69e45f7f3551964617339f98 INFO: 172.20.0.1:52018 - "POST /node/69e45f7f3551964617339f98 HTTP/1.0" 202 Accepted 2026-04-21 04:52:22,956 - lava_callback - INFO - Starting async job submit for node 69e45f7f3551964617339f98 2026-04-21 04:52:22,957 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:52:22,975 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45f7f3551964617339f98 HTTP/1.1" 200 None 2026-04-21 04:52:22,977 - lava_callback - DEBUG - Processing node 69e45f7f3551964617339f98: baseline-arm-clabbe 2026-04-21 04:52:22,977 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e45f7f3551964617339f98 2026-04-21 04:52:22,978 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:52:22,993 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45f7f3551964617339f98 HTTP/1.1" 200 None 2026-04-21 04:52:22,995 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:52:23,018 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e3aefd7f5347282fbdbc78 HTTP/1.1" 200 None 2026-04-21 04:52:23,019 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:52:23,043 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e45f7f3551964617339f98 HTTP/1.1" 200 None 2026-04-21 04:52:23,044 - lava_callback - ERROR - Error processing callback for node 69e45f7f3551964617339f98: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:52032 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52046 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42842 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60198 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40736 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47602 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59392 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59406 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:53:22,987 - lava_callback - INFO - Received callback for node 69e45fac3551964617339fa3 INFO: 172.20.0.1:59362 - "POST /node/69e45fac3551964617339fa3 HTTP/1.0" 202 Accepted 2026-04-21 04:53:22,993 - lava_callback - INFO - Starting async job submit for node 69e45fac3551964617339fa3 2026-04-21 04:53:22,993 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:53:23,008 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45fac3551964617339fa3 HTTP/1.1" 200 None 2026-04-21 04:53:23,009 - lava_callback - DEBUG - Processing node 69e45fac3551964617339fa3: baseline-arm-clabbe 2026-04-21 04:53:23,009 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e45fac3551964617339fa3 2026-04-21 04:53:23,009 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:53:23,024 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45fac3551964617339fa3 HTTP/1.1" 200 None 2026-04-21 04:53:23,025 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:53:23,046 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e3aefd7f5347282fbdbc78 HTTP/1.1" 200 None 2026-04-21 04:53:23,047 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:53:23,064 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e45fac3551964617339fa3 HTTP/1.1" 200 None 2026-04-21 04:53:23,066 - lava_callback - ERROR - Error processing callback for node 69e45fac3551964617339fa3: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:59364 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59380 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50016 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37528 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59314 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37640 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51960 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51968 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:54:23,016 - lava_callback - INFO - Received callback for node 69e45fe93551964617339fb5 INFO: 172.20.0.1:44276 - "POST /node/69e45fe93551964617339fb5 HTTP/1.0" 202 Accepted 2026-04-21 04:54:23,021 - lava_callback - INFO - Starting async job submit for node 69e45fe93551964617339fb5 2026-04-21 04:54:23,022 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:54:23,085 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45fe93551964617339fb5 HTTP/1.1" 200 None 2026-04-21 04:54:23,086 - lava_callback - DEBUG - Processing node 69e45fe93551964617339fb5: baseline-arm-clabbe 2026-04-21 04:54:23,086 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e45fe93551964617339fb5 2026-04-21 04:54:23,087 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:54:23,105 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e45fe93551964617339fb5 HTTP/1.1" 200 None 2026-04-21 04:54:23,106 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:54:23,128 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e3aefd7f5347282fbdbc78 HTTP/1.1" 200 None 2026-04-21 04:54:23,130 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:54:23,167 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e45fe93551964617339fb5 HTTP/1.1" 200 None 2026-04-21 04:54:23,168 - lava_callback - ERROR - Error processing callback for node 69e45fe93551964617339fb5: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:44278 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44288 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54250 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:54:42,995 - lava_callback - INFO - Received callback for node 69e4600e3551964617339fc4 INFO: 172.20.0.1:57620 - "POST /node/69e4600e3551964617339fc4 HTTP/1.0" 202 Accepted 2026-04-21 04:54:43,001 - lava_callback - INFO - Starting async job submit for node 69e4600e3551964617339fc4 2026-04-21 04:54:43,002 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:54:43,017 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4600e3551964617339fc4 HTTP/1.1" 200 None 2026-04-21 04:54:43,018 - lava_callback - DEBUG - Processing node 69e4600e3551964617339fc4: baseline-arm-clabbe 2026-04-21 04:54:43,019 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4600e3551964617339fc4 2026-04-21 04:54:43,019 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:54:43,032 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4600e3551964617339fc4 HTTP/1.1" 200 None 2026-04-21 04:54:43,034 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:54:43,046 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e3aefd7f5347282fbdbc78 HTTP/1.1" 200 None 2026-04-21 04:54:43,047 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:54:43,064 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4600e3551964617339fc4 HTTP/1.1" 200 None 2026-04-21 04:54:43,065 - lava_callback - ERROR - Error processing callback for node 69e4600e3551964617339fc4: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:57634 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51670 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52170 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34414 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34420 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:55:23,091 - lava_callback - INFO - Received callback for node 69e460333551964617339fd7 INFO: 172.20.0.1:52358 - "POST /node/69e460333551964617339fd7 HTTP/1.0" 202 Accepted 2026-04-21 04:55:23,096 - lava_callback - INFO - Starting async job submit for node 69e460333551964617339fd7 2026-04-21 04:55:23,097 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:55:23,121 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e460333551964617339fd7 HTTP/1.1" 200 None 2026-04-21 04:55:23,122 - lava_callback - DEBUG - Processing node 69e460333551964617339fd7: baseline-arm-clabbe 2026-04-21 04:55:23,122 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e460333551964617339fd7 2026-04-21 04:55:23,123 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:55:23,139 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e460333551964617339fd7 HTTP/1.1" 200 None 2026-04-21 04:55:23,141 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:55:23,154 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e3aefd7f5347282fbdbc78 HTTP/1.1" 200 None 2026-04-21 04:55:23,155 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:55:23,172 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e460333551964617339fd7 HTTP/1.1" 200 None 2026-04-21 04:55:23,173 - lava_callback - ERROR - Error processing callback for node 69e460333551964617339fd7: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:52362 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52370 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34534 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53608 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40488 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34774 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45518 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45524 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:56:23,125 - lava_callback - INFO - Received callback for node 69e460613551964617339fe3 INFO: 172.20.0.1:37030 - "POST /node/69e460613551964617339fe3 HTTP/1.0" 202 Accepted 2026-04-21 04:56:23,130 - lava_callback - INFO - Starting async job submit for node 69e460613551964617339fe3 2026-04-21 04:56:23,132 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:56:23,146 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e460613551964617339fe3 HTTP/1.1" 200 None 2026-04-21 04:56:23,146 - lava_callback - DEBUG - Processing node 69e460613551964617339fe3: baseline-arm-clabbe 2026-04-21 04:56:23,147 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e460613551964617339fe3 2026-04-21 04:56:23,147 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:56:23,160 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e460613551964617339fe3 HTTP/1.1" 200 None 2026-04-21 04:56:23,162 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:56:23,175 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e3aefd7f5347282fbdbc78 HTTP/1.1" 200 None 2026-04-21 04:56:23,176 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:56:23,193 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e460613551964617339fe3 HTTP/1.1" 200 None 2026-04-21 04:56:23,194 - lava_callback - ERROR - Error processing callback for node 69e460613551964617339fe3: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:37046 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37062 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52850 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43402 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33986 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:56:58,933 - lava_callback - INFO - Received callback for node 69e460983551964617339ff5 INFO: 172.20.0.1:33990 - "POST /node/69e460983551964617339ff5 HTTP/1.0" 202 Accepted 2026-04-21 04:56:58,939 - lava_callback - INFO - Starting async job submit for node 69e460983551964617339ff5 2026-04-21 04:56:58,940 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:56:58,953 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e460983551964617339ff5 HTTP/1.1" 200 None 2026-04-21 04:56:58,954 - lava_callback - DEBUG - Processing node 69e460983551964617339ff5: baseline-arm64-baylibre 2026-04-21 04:56:58,954 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e460983551964617339ff5 2026-04-21 04:56:58,955 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:56:58,968 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e460983551964617339ff5 HTTP/1.1" 200 None 2026-04-21 04:56:58,969 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:56:58,982 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 04:56:58,983 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:56:59,000 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e460983551964617339ff5 HTTP/1.1" 200 None 2026-04-21 04:56:59,002 - lava_callback - ERROR - Error processing callback for node 69e460983551964617339ff5: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:34144 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:57:11,598 - lava_callback - INFO - Received callback for node 69e678bddc041e577b77b2ee INFO: 172.20.0.1:32796 - "POST /node/69e678bddc041e577b77b2ee HTTP/1.0" 202 Accepted 2026-04-21 04:57:11,604 - lava_callback - INFO - Starting async job submit for node 69e678bddc041e577b77b2ee 2026-04-21 04:57:11,626 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:57:11,642 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e678bddc041e577b77b2ee HTTP/1.1" 200 None 2026-04-21 04:57:11,643 - lava_callback - DEBUG - Processing node 69e678bddc041e577b77b2ee: baseline-arm64-broonie 2026-04-21 04:57:11,864 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 04:57:12,148 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 04:57:12,149 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e678bddc041e577b77b2ee/log.txt.gz 2026-04-21 04:57:12,298 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 04:57:12,448 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 04:57:12,450 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e678bddc041e577b77b2ee/lava-logs.yaml 2026-04-21 04:57:12,456 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 04:57:12,578 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 04:57:12,580 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e678bddc041e577b77b2ee/lava_callback.json.gz 2026-04-21 04:57:12,607 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:57:12,626 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e678bddc041e577b77b2ee HTTP/1.1" 200 None 2026-04-21 04:57:12,628 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:57:12,644 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e66a77dc041e577b7785d5 HTTP/1.1" 200 None 2026-04-21 04:57:12,646 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:57:12,675 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e678bddc041e577b77b2ee HTTP/1.1" 200 None 2026-04-21 04:57:12,677 - lava_callback - INFO - Completed processing callback for node 69e678bddc041e577b77b2ee INFO: 172.20.0.1:32806 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 04:57:16,030 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:57:16,050 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:32810 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60616 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60618 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55662 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50996 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38398 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39094 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:58:11,180 - lava_callback - INFO - Received callback for node 69e6923fdc041e577b77c0f1 INFO: 172.20.0.1:58838 - "POST /node/69e6923fdc041e577b77c0f1 HTTP/1.0" 202 Accepted 2026-04-21 04:58:11,186 - lava_callback - INFO - Starting async job submit for node 69e6923fdc041e577b77c0f1 2026-04-21 04:58:11,209 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:58:11,222 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6923fdc041e577b77c0f1 HTTP/1.1" 200 None 2026-04-21 04:58:11,223 - lava_callback - DEBUG - Processing node 69e6923fdc041e577b77c0f1: baseline-arm64-broonie 2026-04-21 04:58:11,447 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 04:58:11,528 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 04:58:11,530 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6923fdc041e577b77c0f1/log.txt.gz 2026-04-21 04:58:11,661 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 04:58:11,948 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 04:58:11,950 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6923fdc041e577b77c0f1/lava-logs.yaml 2026-04-21 04:58:11,957 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 04:58:12,073 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 04:58:12,076 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6923fdc041e577b77c0f1/lava_callback.json.gz 2026-04-21 04:58:12,111 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:58:12,124 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6923fdc041e577b77c0f1 HTTP/1.1" 200 None 2026-04-21 04:58:12,126 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:58:12,142 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6522047863bb428fe4af5 HTTP/1.1" 200 None 2026-04-21 04:58:12,143 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:58:12,165 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6923fdc041e577b77c0f1 HTTP/1.1" 200 None 2026-04-21 04:58:12,167 - lava_callback - INFO - Completed processing callback for node 69e6923fdc041e577b77c0f1 INFO: 172.20.0.1:58844 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 04:58:16,053 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:58:16,069 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:58850 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:58:23,152 - lava_callback - INFO - Received callback for node 69e460e3355196461733a017 INFO: 172.20.0.1:51034 - "POST /node/69e460e3355196461733a017 HTTP/1.0" 202 Accepted 2026-04-21 04:58:23,158 - lava_callback - INFO - Starting async job submit for node 69e460e3355196461733a017 2026-04-21 04:58:23,159 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:58:23,173 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e460e3355196461733a017 HTTP/1.1" 200 None 2026-04-21 04:58:23,174 - lava_callback - DEBUG - Processing node 69e460e3355196461733a017: baseline-arm-clabbe 2026-04-21 04:58:23,174 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e460e3355196461733a017 2026-04-21 04:58:23,174 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:58:23,187 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e460e3355196461733a017 HTTP/1.1" 200 None 2026-04-21 04:58:23,189 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:58:23,201 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e3aefd7f5347282fbdbc78 HTTP/1.1" 200 None 2026-04-21 04:58:23,203 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:58:23,220 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e460e3355196461733a017 HTTP/1.1" 200 None 2026-04-21 04:58:23,221 - lava_callback - ERROR - Error processing callback for node 69e460e3355196461733a017: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:51038 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51040 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:58:31,499 - lava_callback - INFO - Received callback for node 69e6cb9860f5826562c15c64 INFO: 172.20.0.1:45906 - "POST /node/69e6cb9860f5826562c15c64 HTTP/1.0" 202 Accepted 2026-04-21 04:58:31,505 - lava_callback - INFO - Starting async job submit for node 69e6cb9860f5826562c15c64 2026-04-21 04:58:31,527 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:58:31,543 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb9860f5826562c15c64 HTTP/1.1" 200 None 2026-04-21 04:58:31,544 - lava_callback - DEBUG - Processing node 69e6cb9860f5826562c15c64: baseline-arm64-broonie 2026-04-21 04:58:31,769 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 04:58:32,060 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 04:58:32,062 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cb9860f5826562c15c64/log.txt.gz 2026-04-21 04:58:32,234 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 04:58:32,402 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 04:58:32,404 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cb9860f5826562c15c64/lava-logs.yaml 2026-04-21 04:58:32,410 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 04:58:32,535 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 04:58:32,537 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cb9860f5826562c15c64/lava_callback.json.gz 2026-04-21 04:58:32,561 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:58:32,596 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb9860f5826562c15c64 HTTP/1.1" 200 None 2026-04-21 04:58:32,598 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:58:32,611 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 04:58:32,613 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:58:32,636 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb9860f5826562c15c64 HTTP/1.1" 200 None 2026-04-21 04:58:32,638 - lava_callback - INFO - Completed processing callback for node 69e6cb9860f5826562c15c64 INFO: 172.20.0.1:45910 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:58:46,072 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:58:46,090 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:43152 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52184 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:58:58,758 - lava_callback - INFO - Received callback for node 69e4610d355196461733a029 INFO: 172.20.0.1:52200 - "POST /node/69e4610d355196461733a029 HTTP/1.0" 202 Accepted 2026-04-21 04:58:58,763 - lava_callback - INFO - Starting async job submit for node 69e4610d355196461733a029 2026-04-21 04:58:58,764 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:58:58,779 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4610d355196461733a029 HTTP/1.1" 200 None 2026-04-21 04:58:58,780 - lava_callback - DEBUG - Processing node 69e4610d355196461733a029: baseline-arm64-baylibre 2026-04-21 04:58:58,780 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4610d355196461733a029 2026-04-21 04:58:58,780 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:58:58,793 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4610d355196461733a029 HTTP/1.1" 200 None 2026-04-21 04:58:58,795 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:58:58,808 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 04:58:58,809 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:58:58,827 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4610d355196461733a029 HTTP/1.1" 200 None 2026-04-21 04:58:58,828 - lava_callback - ERROR - Error processing callback for node 69e4610d355196461733a029: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:58582 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36006 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:36018 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58766 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58770 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53614 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35288 - "GET / HTTP/1.0" 200 OK 2026-04-21 04:59:52,045 - lava_callback - INFO - Received callback for node 69e6d4d060f5826562c16247 INFO: 172.20.0.1:59504 - "POST /node/69e6d4d060f5826562c16247 HTTP/1.0" 202 Accepted 2026-04-21 04:59:52,051 - lava_callback - INFO - Starting async job submit for node 69e6d4d060f5826562c16247 2026-04-21 04:59:52,074 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:59:52,088 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6d4d060f5826562c16247 HTTP/1.1" 200 None 2026-04-21 04:59:52,089 - lava_callback - DEBUG - Processing node 69e6d4d060f5826562c16247: baseline-arm64-broonie 2026-04-21 04:59:52,351 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 04:59:52,494 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 04:59:52,496 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6d4d060f5826562c16247/log.txt.gz 2026-04-21 04:59:52,628 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 04:59:52,827 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 04:59:52,828 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6d4d060f5826562c16247/lava-logs.yaml 2026-04-21 04:59:52,835 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 04:59:52,957 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 04:59:52,958 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6d4d060f5826562c16247/lava_callback.json.gz 2026-04-21 04:59:52,981 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:59:52,995 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6d4d060f5826562c16247 HTTP/1.1" 200 None 2026-04-21 04:59:52,996 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:59:53,012 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e66a77dc041e577b7785d5 HTTP/1.1" 200 None 2026-04-21 04:59:53,013 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 04:59:53,034 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6d4d060f5826562c16247 HTTP/1.1" 200 None 2026-04-21 04:59:53,035 - lava_callback - INFO - Completed processing callback for node 69e6d4d060f5826562c16247 INFO: 172.20.0.1:59520 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59622 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:00:11,181 - lava_callback - INFO - Received callback for node 69e6eb6460f5826562c16d24 INFO: 172.20.0.1:58940 - "POST /node/69e6eb6460f5826562c16d24 HTTP/1.0" 202 Accepted 2026-04-21 05:00:11,188 - lava_callback - INFO - Starting async job submit for node 69e6eb6460f5826562c16d24 2026-04-21 05:00:11,211 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:00:11,225 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6eb6460f5826562c16d24 HTTP/1.1" 200 None 2026-04-21 05:00:11,226 - lava_callback - DEBUG - Processing node 69e6eb6460f5826562c16d24: baseline-arm64-broonie 2026-04-21 05:00:11,456 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 05:00:11,540 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 05:00:11,542 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6eb6460f5826562c16d24/log.txt.gz 2026-04-21 05:00:11,674 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 05:00:11,982 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 05:00:11,984 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6eb6460f5826562c16d24/lava-logs.yaml 2026-04-21 05:00:11,990 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 05:00:12,114 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 05:00:12,116 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6eb6460f5826562c16d24/lava_callback.json.gz 2026-04-21 05:00:12,138 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:00:12,152 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6eb6460f5826562c16d24 HTTP/1.1" 200 None 2026-04-21 05:00:12,153 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:00:12,166 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6522047863bb428fe4af5 HTTP/1.1" 200 None 2026-04-21 05:00:12,167 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:00:12,187 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6eb6460f5826562c16d24 HTTP/1.1" 200 None 2026-04-21 05:00:12,188 - lava_callback - INFO - Completed processing callback for node 69e6eb6460f5826562c16d24 INFO: 172.20.0.1:58948 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 05:00:16,092 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:00:16,109 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:58954 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:00:18,873 - lava_callback - INFO - Received callback for node 69e4615f355196461733a04e INFO: 172.20.0.1:58966 - "POST /node/69e4615f355196461733a04e HTTP/1.0" 202 Accepted 2026-04-21 05:00:18,879 - lava_callback - INFO - Starting async job submit for node 69e4615f355196461733a04e 2026-04-21 05:00:18,880 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:00:18,894 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4615f355196461733a04e HTTP/1.1" 200 None 2026-04-21 05:00:18,895 - lava_callback - DEBUG - Processing node 69e4615f355196461733a04e: baseline-arm64-baylibre 2026-04-21 05:00:18,895 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4615f355196461733a04e 2026-04-21 05:00:18,896 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:00:18,909 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4615f355196461733a04e HTTP/1.1" 200 None 2026-04-21 05:00:18,910 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:00:18,923 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:00:18,924 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:00:18,945 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4615f355196461733a04e HTTP/1.1" 200 None 2026-04-21 05:00:18,946 - lava_callback - ERROR - Error processing callback for node 69e4615f355196461733a04e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:43702 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43712 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46056 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58868 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35860 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43458 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47496 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47498 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42396 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42410 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41232 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60868 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60434 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59838 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41648 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41658 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47080 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47092 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43090 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49550 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50354 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:02:59,187 - lava_callback - INFO - Received callback for node 69e461f1355196461733a073 INFO: 172.20.0.1:50368 - "POST /node/69e461f1355196461733a073 HTTP/1.0" 202 Accepted 2026-04-21 05:02:59,193 - lava_callback - INFO - Starting async job submit for node 69e461f1355196461733a073 2026-04-21 05:02:59,194 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:02:59,216 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e461f1355196461733a073 HTTP/1.1" 200 None 2026-04-21 05:02:59,217 - lava_callback - DEBUG - Processing node 69e461f1355196461733a073: baseline-arm64-baylibre 2026-04-21 05:02:59,217 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e461f1355196461733a073 2026-04-21 05:02:59,218 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:02:59,230 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e461f1355196461733a073 HTTP/1.1" 200 None 2026-04-21 05:02:59,232 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:02:59,245 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:02:59,246 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:02:59,264 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e461f1355196461733a073 HTTP/1.1" 200 None 2026-04-21 05:02:59,265 - lava_callback - ERROR - Error processing callback for node 69e461f1355196461733a073: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:55032 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46502 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46518 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54906 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54922 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47752 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34794 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50464 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52076 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60328 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60330 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50334 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50338 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35676 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47392 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40222 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50352 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48614 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48620 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:05:19,199 - lava_callback - INFO - Received callback for node 69e4628a355196461733a0be INFO: 172.20.0.1:48632 - "POST /node/69e4628a355196461733a0be HTTP/1.0" 202 Accepted 2026-04-21 05:05:19,205 - lava_callback - INFO - Starting async job submit for node 69e4628a355196461733a0be 2026-04-21 05:05:19,206 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:05:19,221 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4628a355196461733a0be HTTP/1.1" 200 None 2026-04-21 05:05:19,222 - lava_callback - DEBUG - Processing node 69e4628a355196461733a0be: baseline-arm64-baylibre 2026-04-21 05:05:19,222 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4628a355196461733a0be 2026-04-21 05:05:19,223 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:05:19,235 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4628a355196461733a0be HTTP/1.1" 200 None 2026-04-21 05:05:19,237 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:05:19,250 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:05:19,251 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:05:19,268 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4628a355196461733a0be HTTP/1.1" 200 None 2026-04-21 05:05:19,269 - lava_callback - ERROR - Error processing callback for node 69e4628a355196461733a0be: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:38618 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38634 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54824 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47120 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53694 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57090 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54994 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:55002 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51112 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51126 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43752 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:06:39,217 - lava_callback - INFO - Received callback for node 69e462d4355196461733a0dd INFO: 172.20.0.1:43758 - "POST /node/69e462d4355196461733a0dd HTTP/1.0" 202 Accepted 2026-04-21 05:06:39,223 - lava_callback - INFO - Starting async job submit for node 69e462d4355196461733a0dd 2026-04-21 05:06:39,224 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:06:39,238 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e462d4355196461733a0dd HTTP/1.1" 200 None 2026-04-21 05:06:39,239 - lava_callback - DEBUG - Processing node 69e462d4355196461733a0dd: baseline-arm64-baylibre 2026-04-21 05:06:39,239 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e462d4355196461733a0dd 2026-04-21 05:06:39,240 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:06:39,253 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e462d4355196461733a0dd HTTP/1.1" 200 None 2026-04-21 05:06:39,255 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:06:39,271 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:06:39,272 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:06:39,291 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e462d4355196461733a0dd HTTP/1.1" 200 None 2026-04-21 05:06:39,292 - lava_callback - ERROR - Error processing callback for node 69e462d4355196461733a0dd: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:60436 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34206 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46180 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50396 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50408 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45518 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45524 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47496 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55146 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48880 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:07:59,275 - lava_callback - INFO - Received callback for node 69e46326355196461733a103 INFO: 172.20.0.1:48888 - "POST /node/69e46326355196461733a103 HTTP/1.0" 202 Accepted 2026-04-21 05:07:59,280 - lava_callback - INFO - Starting async job submit for node 69e46326355196461733a103 2026-04-21 05:07:59,281 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:07:59,295 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46326355196461733a103 HTTP/1.1" 200 None 2026-04-21 05:07:59,296 - lava_callback - DEBUG - Processing node 69e46326355196461733a103: baseline-arm64-baylibre 2026-04-21 05:07:59,296 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e46326355196461733a103 2026-04-21 05:07:59,297 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:07:59,320 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46326355196461733a103 HTTP/1.1" 200 None 2026-04-21 05:07:59,322 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:07:59,336 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:07:59,337 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:07:59,354 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e46326355196461733a103 HTTP/1.1" 200 None 2026-04-21 05:07:59,356 - lava_callback - ERROR - Error processing callback for node 69e46326355196461733a103: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:34752 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49654 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49664 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38422 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38432 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52800 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38086 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49030 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45930 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37486 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37498 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:09:19,329 - lava_callback - INFO - Received callback for node 69e46371355196461733a126 INFO: 172.20.0.1:37510 - "POST /node/69e46371355196461733a126 HTTP/1.0" 202 Accepted 2026-04-21 05:09:19,335 - lava_callback - INFO - Starting async job submit for node 69e46371355196461733a126 2026-04-21 05:09:19,336 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:09:19,356 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46371355196461733a126 HTTP/1.1" 200 None 2026-04-21 05:09:19,357 - lava_callback - DEBUG - Processing node 69e46371355196461733a126: baseline-arm64-baylibre 2026-04-21 05:09:19,357 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e46371355196461733a126 2026-04-21 05:09:19,357 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:09:19,372 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46371355196461733a126 HTTP/1.1" 200 None 2026-04-21 05:09:19,374 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:09:19,390 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:09:19,392 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:09:19,410 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e46371355196461733a126 HTTP/1.1" 200 None 2026-04-21 05:09:19,411 - lava_callback - ERROR - Error processing callback for node 69e46371355196461733a126: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:59274 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59288 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44514 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57170 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37220 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56046 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34460 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34474 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55132 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55146 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49748 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:10:39,401 - lava_callback - INFO - Received callback for node 69e463c0355196461733a14b INFO: 172.20.0.1:49756 - "POST /node/69e463c0355196461733a14b HTTP/1.0" 202 Accepted 2026-04-21 05:10:39,406 - lava_callback - INFO - Starting async job submit for node 69e463c0355196461733a14b 2026-04-21 05:10:39,407 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:10:39,424 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e463c0355196461733a14b HTTP/1.1" 200 None 2026-04-21 05:10:39,424 - lava_callback - DEBUG - Processing node 69e463c0355196461733a14b: baseline-arm64-baylibre 2026-04-21 05:10:39,425 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e463c0355196461733a14b 2026-04-21 05:10:39,425 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:10:39,438 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e463c0355196461733a14b HTTP/1.1" 200 None 2026-04-21 05:10:39,439 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:10:39,452 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:10:39,453 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:10:39,471 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e463c0355196461733a14b HTTP/1.1" 200 None 2026-04-21 05:10:39,472 - lava_callback - ERROR - Error processing callback for node 69e463c0355196461733a14b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:52220 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58906 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41464 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48790 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48802 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48308 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48322 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39402 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56094 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32902 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45966 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44792 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44796 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32846 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32856 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37078 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37800 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42592 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54028 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59608 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59622 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36084 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36086 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35466 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:13:43,891 - lava_callback - INFO - Received callback for node 69e46475355196461733a18e INFO: 172.20.0.1:53144 - "POST /node/69e46475355196461733a18e HTTP/1.0" 202 Accepted 2026-04-21 05:13:43,897 - lava_callback - INFO - Starting async job submit for node 69e46475355196461733a18e 2026-04-21 05:13:43,898 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:13:43,912 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46475355196461733a18e HTTP/1.1" 200 None 2026-04-21 05:13:43,912 - lava_callback - DEBUG - Processing node 69e46475355196461733a18e: baseline-arm64-clabbe 2026-04-21 05:13:43,913 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e46475355196461733a18e 2026-04-21 05:13:43,913 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:13:43,926 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46475355196461733a18e HTTP/1.1" 200 None 2026-04-21 05:13:43,927 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:13:43,947 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:13:43,949 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:13:43,971 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e46475355196461733a18e HTTP/1.1" 200 None 2026-04-21 05:13:43,972 - lava_callback - ERROR - Error processing callback for node 69e46475355196461733a18e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:53156 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52730 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36562 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40884 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40894 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47108 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47110 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40756 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40768 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35934 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:15:03,916 - lava_callback - INFO - Received callback for node 69e464c2355196461733a1ac INFO: 172.20.0.1:49088 - "POST /node/69e464c2355196461733a1ac HTTP/1.0" 202 Accepted 2026-04-21 05:15:03,922 - lava_callback - INFO - Starting async job submit for node 69e464c2355196461733a1ac 2026-04-21 05:15:03,923 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:15:03,948 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e464c2355196461733a1ac HTTP/1.1" 200 None 2026-04-21 05:15:03,949 - lava_callback - DEBUG - Processing node 69e464c2355196461733a1ac: baseline-arm64-clabbe 2026-04-21 05:15:03,949 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e464c2355196461733a1ac 2026-04-21 05:15:03,950 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:15:03,963 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e464c2355196461733a1ac HTTP/1.1" 200 None 2026-04-21 05:15:03,965 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:15:03,978 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:15:03,979 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:15:03,997 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e464c2355196461733a1ac HTTP/1.1" 200 None 2026-04-21 05:15:03,998 - lava_callback - ERROR - Error processing callback for node 69e464c2355196461733a1ac: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49102 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44140 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44156 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52112 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52124 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35998 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52118 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37158 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:15:59,622 - lava_callback - INFO - Received callback for node 69e46509355196461733a1cc INFO: 172.20.0.1:37166 - "POST /node/69e46509355196461733a1cc HTTP/1.0" 202 Accepted 2026-04-21 05:15:59,628 - lava_callback - INFO - Starting async job submit for node 69e46509355196461733a1cc 2026-04-21 05:15:59,629 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:15:59,643 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46509355196461733a1cc HTTP/1.1" 200 None 2026-04-21 05:15:59,644 - lava_callback - DEBUG - Processing node 69e46509355196461733a1cc: baseline-arm64-baylibre 2026-04-21 05:15:59,644 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e46509355196461733a1cc 2026-04-21 05:15:59,645 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:15:59,657 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46509355196461733a1cc HTTP/1.1" 200 None 2026-04-21 05:15:59,659 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:15:59,671 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:15:59,672 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:15:59,690 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e46509355196461733a1cc HTTP/1.1" 200 None 2026-04-21 05:15:59,691 - lava_callback - ERROR - Error processing callback for node 69e46509355196461733a1cc: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54082 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43332 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43336 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58366 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58376 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55714 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50300 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60690 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50770 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37862 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37870 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:17:19,730 - lava_callback - INFO - Received callback for node 69e46556355196461733a1ed INFO: 172.20.0.1:55778 - "POST /node/69e46556355196461733a1ed HTTP/1.0" 202 Accepted 2026-04-21 05:17:19,736 - lava_callback - INFO - Starting async job submit for node 69e46556355196461733a1ed 2026-04-21 05:17:19,737 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:17:19,751 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46556355196461733a1ed HTTP/1.1" 200 None 2026-04-21 05:17:19,752 - lava_callback - DEBUG - Processing node 69e46556355196461733a1ed: baseline-arm64-baylibre 2026-04-21 05:17:19,752 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e46556355196461733a1ed 2026-04-21 05:17:19,753 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:17:19,767 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46556355196461733a1ed HTTP/1.1" 200 None 2026-04-21 05:17:19,769 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:17:19,782 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:17:19,783 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:17:19,800 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e46556355196461733a1ed HTTP/1.1" 200 None 2026-04-21 05:17:19,801 - lava_callback - ERROR - Error processing callback for node 69e46556355196461733a1ed: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:55788 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55800 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45694 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37740 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54646 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34592 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38144 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:38154 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47032 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47040 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41006 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:18:44,080 - lava_callback - INFO - Received callback for node 69e465a9355196461733a210 INFO: 172.20.0.1:59458 - "POST /node/69e465a9355196461733a210 HTTP/1.0" 202 Accepted 2026-04-21 05:18:44,086 - lava_callback - INFO - Starting async job submit for node 69e465a9355196461733a210 2026-04-21 05:18:44,087 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:18:44,105 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e465a9355196461733a210 HTTP/1.1" 200 None 2026-04-21 05:18:44,106 - lava_callback - DEBUG - Processing node 69e465a9355196461733a210: baseline-arm64-clabbe 2026-04-21 05:18:44,106 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e465a9355196461733a210 2026-04-21 05:18:44,107 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:18:44,120 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e465a9355196461733a210 HTTP/1.1" 200 None 2026-04-21 05:18:44,121 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:18:44,134 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:18:44,135 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:18:44,159 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e465a9355196461733a210 HTTP/1.1" 200 None 2026-04-21 05:18:44,160 - lava_callback - ERROR - Error processing callback for node 69e465a9355196461733a210: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:59462 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54248 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38920 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53710 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53720 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33708 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33716 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33398 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49330 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37730 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:19:59,888 - lava_callback - INFO - Received callback for node 69e465f0355196461733a231 INFO: 172.20.0.1:56752 - "POST /node/69e465f0355196461733a231 HTTP/1.0" 202 Accepted 2026-04-21 05:19:59,894 - lava_callback - INFO - Starting async job submit for node 69e465f0355196461733a231 2026-04-21 05:19:59,895 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:19:59,909 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e465f0355196461733a231 HTTP/1.1" 200 None 2026-04-21 05:19:59,910 - lava_callback - DEBUG - Processing node 69e465f0355196461733a231: baseline-arm64-baylibre 2026-04-21 05:19:59,910 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e465f0355196461733a231 2026-04-21 05:19:59,911 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:19:59,930 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e465f0355196461733a231 HTTP/1.1" 200 None 2026-04-21 05:19:59,931 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:19:59,944 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:19:59,946 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:19:59,963 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e465f0355196461733a231 HTTP/1.1" 200 None 2026-04-21 05:19:59,964 - lava_callback - ERROR - Error processing callback for node 69e465f0355196461733a231: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:56756 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47352 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47368 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36720 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36724 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53638 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50118 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48748 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56678 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40298 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40300 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:21:24,183 - lava_callback - INFO - Received callback for node 69e46643355196461733a252 INFO: 172.20.0.1:53352 - "POST /node/69e46643355196461733a252 HTTP/1.0" 202 Accepted 2026-04-21 05:21:24,188 - lava_callback - INFO - Starting async job submit for node 69e46643355196461733a252 2026-04-21 05:21:24,189 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:21:24,204 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46643355196461733a252 HTTP/1.1" 200 None 2026-04-21 05:21:24,205 - lava_callback - DEBUG - Processing node 69e46643355196461733a252: baseline-arm64-clabbe 2026-04-21 05:21:24,205 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e46643355196461733a252 2026-04-21 05:21:24,205 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:21:24,218 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46643355196461733a252 HTTP/1.1" 200 None 2026-04-21 05:21:24,219 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:21:24,235 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:21:24,236 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:21:24,254 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e46643355196461733a252 HTTP/1.1" 200 None 2026-04-21 05:21:24,255 - lava_callback - ERROR - Error processing callback for node 69e46643355196461733a252: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:53364 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53376 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38702 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59716 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47180 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35284 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57410 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57424 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58366 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58376 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43308 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:22:44,230 - lava_callback - INFO - Received callback for node 69e46691355196461733a279 INFO: 172.20.0.1:43960 - "POST /node/69e46691355196461733a279 HTTP/1.0" 202 Accepted 2026-04-21 05:22:44,235 - lava_callback - INFO - Starting async job submit for node 69e46691355196461733a279 2026-04-21 05:22:44,236 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:22:44,250 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46691355196461733a279 HTTP/1.1" 200 None 2026-04-21 05:22:44,251 - lava_callback - DEBUG - Processing node 69e46691355196461733a279: baseline-arm64-clabbe 2026-04-21 05:22:44,251 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e46691355196461733a279 2026-04-21 05:22:44,251 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:22:44,264 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46691355196461733a279 HTTP/1.1" 200 None 2026-04-21 05:22:44,266 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:22:44,279 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:22:44,280 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:22:44,297 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e46691355196461733a279 HTTP/1.1" 200 None 2026-04-21 05:22:44,298 - lava_callback - ERROR - Error processing callback for node 69e46691355196461733a279: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:43972 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59210 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33044 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37682 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37698 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49316 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49324 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45264 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:23:40,190 - lava_callback - INFO - Received callback for node 69e466d8355196461733a298 INFO: 172.20.0.1:42872 - "POST /node/69e466d8355196461733a298 HTTP/1.0" 202 Accepted 2026-04-21 05:23:40,197 - lava_callback - INFO - Starting async job submit for node 69e466d8355196461733a298 2026-04-21 05:23:40,198 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:23:40,213 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e466d8355196461733a298 HTTP/1.1" 200 None 2026-04-21 05:23:40,213 - lava_callback - DEBUG - Processing node 69e466d8355196461733a298: baseline-arm64-baylibre 2026-04-21 05:23:40,214 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e466d8355196461733a298 2026-04-21 05:23:40,214 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:23:40,228 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e466d8355196461733a298 HTTP/1.1" 200 None 2026-04-21 05:23:40,229 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:23:40,250 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:23:40,252 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:23:40,270 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e466d8355196461733a298 HTTP/1.1" 200 None 2026-04-21 05:23:40,271 - lava_callback - ERROR - Error processing callback for node 69e466d8355196461733a298: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:42878 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55700 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57218 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51868 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51880 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43708 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43716 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40398 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36102 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44898 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:25:04,395 - lava_callback - INFO - Received callback for node 69e46728355196461733a2be INFO: 172.20.0.1:50266 - "POST /node/69e46728355196461733a2be HTTP/1.0" 202 Accepted 2026-04-21 05:25:04,401 - lava_callback - INFO - Starting async job submit for node 69e46728355196461733a2be 2026-04-21 05:25:04,402 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:25:04,418 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46728355196461733a2be HTTP/1.1" 200 None 2026-04-21 05:25:04,419 - lava_callback - DEBUG - Processing node 69e46728355196461733a2be: baseline-arm64-clabbe 2026-04-21 05:25:04,419 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e46728355196461733a2be 2026-04-21 05:25:04,420 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:25:04,435 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46728355196461733a2be HTTP/1.1" 200 None 2026-04-21 05:25:04,437 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:25:04,449 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:25:04,451 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:25:04,470 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e46728355196461733a2be HTTP/1.1" 200 None 2026-04-21 05:25:04,472 - lava_callback - ERROR - Error processing callback for node 69e46728355196461733a2be: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:50274 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48532 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48542 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39160 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39176 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49788 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54620 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40664 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41366 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33038 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33046 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:26:20,183 - lava_callback - INFO - Received callback for node 69e4676f355196461733a2dc INFO: 172.20.0.1:44430 - "POST /node/69e4676f355196461733a2dc HTTP/1.0" 202 Accepted 2026-04-21 05:26:20,189 - lava_callback - INFO - Starting async job submit for node 69e4676f355196461733a2dc 2026-04-21 05:26:20,190 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:26:20,206 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4676f355196461733a2dc HTTP/1.1" 200 None 2026-04-21 05:26:20,207 - lava_callback - DEBUG - Processing node 69e4676f355196461733a2dc: baseline-arm64-baylibre 2026-04-21 05:26:20,207 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4676f355196461733a2dc 2026-04-21 05:26:20,208 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:26:20,221 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4676f355196461733a2dc HTTP/1.1" 200 None 2026-04-21 05:26:20,222 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:26:20,235 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:26:20,236 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:26:20,263 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4676f355196461733a2dc HTTP/1.1" 200 None 2026-04-21 05:26:20,267 - lava_callback - ERROR - Error processing callback for node 69e4676f355196461733a2dc: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:44446 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44448 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35418 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56054 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44122 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52386 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49622 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49634 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46616 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46626 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47384 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:27:44,437 - lava_callback - INFO - Received callback for node 69e467c1355196461733a300 INFO: 172.20.0.1:39500 - "POST /node/69e467c1355196461733a300 HTTP/1.0" 202 Accepted 2026-04-21 05:27:44,443 - lava_callback - INFO - Starting async job submit for node 69e467c1355196461733a300 2026-04-21 05:27:44,444 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:27:44,460 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e467c1355196461733a300 HTTP/1.1" 200 None 2026-04-21 05:27:44,461 - lava_callback - DEBUG - Processing node 69e467c1355196461733a300: baseline-arm64-clabbe 2026-04-21 05:27:44,461 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e467c1355196461733a300 2026-04-21 05:27:44,462 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:27:44,474 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e467c1355196461733a300 HTTP/1.1" 200 None 2026-04-21 05:27:44,476 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:27:44,493 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:27:44,494 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:27:44,512 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e467c1355196461733a300 HTTP/1.1" 200 None 2026-04-21 05:27:44,513 - lava_callback - ERROR - Error processing callback for node 69e467c1355196461733a300: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:39504 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37938 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33292 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51934 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51938 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55802 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55804 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55502 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34908 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58248 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:29:00,447 - lava_callback - INFO - Received callback for node 69e46809355196461733a31f INFO: 172.20.0.1:55586 - "POST /node/69e46809355196461733a31f HTTP/1.0" 202 Accepted 2026-04-21 05:29:00,452 - lava_callback - INFO - Starting async job submit for node 69e46809355196461733a31f 2026-04-21 05:29:00,453 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:29:00,468 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46809355196461733a31f HTTP/1.1" 200 None 2026-04-21 05:29:00,469 - lava_callback - DEBUG - Processing node 69e46809355196461733a31f: baseline-arm64-baylibre 2026-04-21 05:29:00,469 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e46809355196461733a31f 2026-04-21 05:29:00,469 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:29:00,482 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46809355196461733a31f HTTP/1.1" 200 None 2026-04-21 05:29:00,484 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:29:00,497 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:29:00,498 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:29:00,517 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e46809355196461733a31f HTTP/1.1" 200 None 2026-04-21 05:29:00,518 - lava_callback - ERROR - Error processing callback for node 69e46809355196461733a31f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:55602 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49000 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49010 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35790 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35806 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53520 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58928 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49354 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:30:04,706 - lava_callback - INFO - Received callback for node 69e46859355196461733a343 INFO: 172.20.0.1:46332 - "POST /node/69e46859355196461733a343 HTTP/1.0" 202 Accepted 2026-04-21 05:30:04,712 - lava_callback - INFO - Starting async job submit for node 69e46859355196461733a343 2026-04-21 05:30:04,713 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:30:04,727 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46859355196461733a343 HTTP/1.1" 200 None 2026-04-21 05:30:04,728 - lava_callback - DEBUG - Processing node 69e46859355196461733a343: baseline-arm64-clabbe 2026-04-21 05:30:04,728 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e46859355196461733a343 2026-04-21 05:30:04,729 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:30:04,741 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46859355196461733a343 HTTP/1.1" 200 None 2026-04-21 05:30:04,742 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:30:04,756 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:30:04,758 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:30:04,782 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e46859355196461733a343 HTTP/1.1" 200 None 2026-04-21 05:30:04,783 - lava_callback - ERROR - Error processing callback for node 69e46859355196461733a343: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:46338 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36802 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 05:30:15,853 - lava_callback - INFO - Received callback for node 69e6eea160f5826562c16e94 INFO: 172.20.0.1:36804 - "POST /node/69e6eea160f5826562c16e94 HTTP/1.0" 202 Accepted 2026-04-21 05:30:15,860 - lava_callback - INFO - Starting async job submit for node 69e6eea160f5826562c16e94 2026-04-21 05:30:15,884 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:30:15,900 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6eea160f5826562c16e94 HTTP/1.1" 200 None 2026-04-21 05:30:15,901 - lava_callback - DEBUG - Processing node 69e6eea160f5826562c16e94: baseline-arm64 2026-04-21 05:30:16,489 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.20.0.1:36812 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:30:16,657 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 05:30:16,658 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6eea160f5826562c16e94/log.txt.gz 2026-04-21 05:30:16,902 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 05:30:17,293 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 05:30:17,295 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6eea160f5826562c16e94/lava-logs.yaml 2026-04-21 05:30:17,307 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 05:30:17,462 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 05:30:17,464 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69e6eea160f5826562c16e94/lava_callback.json.gz 2026-04-21 05:30:17,488 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:30:17,509 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6eea160f5826562c16e94 HTTP/1.1" 200 None 2026-04-21 05:30:17,510 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:30:17,529 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 05:30:17,530 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:30:17,552 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6eea160f5826562c16e94 HTTP/1.1" 200 None 2026-04-21 05:30:17,608 - lava_callback - INFO - Completed processing callback for node 69e6eea160f5826562c16e94 INFO: 172.20.0.1:36102 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36114 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47024 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:30:46,122 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:30:46,139 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:53702 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59266 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37908 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34950 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34964 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:31:20,472 - lava_callback - INFO - Received callback for node 69e4689f355196461733a364 INFO: 172.20.0.1:39832 - "POST /node/69e4689f355196461733a364 HTTP/1.0" 202 Accepted 2026-04-21 05:31:20,477 - lava_callback - INFO - Starting async job submit for node 69e4689f355196461733a364 2026-04-21 05:31:20,478 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:31:20,493 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4689f355196461733a364 HTTP/1.1" 200 None 2026-04-21 05:31:20,493 - lava_callback - DEBUG - Processing node 69e4689f355196461733a364: baseline-arm64-baylibre 2026-04-21 05:31:20,494 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4689f355196461733a364 2026-04-21 05:31:20,494 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:31:20,507 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4689f355196461733a364 HTTP/1.1" 200 None 2026-04-21 05:31:20,509 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:31:20,521 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:31:20,523 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:31:20,540 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4689f355196461733a364 HTTP/1.1" 200 None 2026-04-21 05:31:20,541 - lava_callback - ERROR - Error processing callback for node 69e4689f355196461733a364: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:39840 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39854 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46716 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53090 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55588 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49434 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59882 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59898 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51730 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51738 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41376 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:32:44,670 - lava_callback - INFO - Received callback for node 69e468ef355196461733a385 INFO: 172.20.0.1:59080 - "POST /node/69e468ef355196461733a385 HTTP/1.0" 202 Accepted 2026-04-21 05:32:44,676 - lava_callback - INFO - Starting async job submit for node 69e468ef355196461733a385 2026-04-21 05:32:44,677 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:32:44,692 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e468ef355196461733a385 HTTP/1.1" 200 None 2026-04-21 05:32:44,693 - lava_callback - DEBUG - Processing node 69e468ef355196461733a385: baseline-arm64-clabbe 2026-04-21 05:32:44,693 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e468ef355196461733a385 2026-04-21 05:32:44,694 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:32:44,707 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e468ef355196461733a385 HTTP/1.1" 200 None 2026-04-21 05:32:44,708 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:32:44,720 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:32:44,722 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:32:44,739 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e468ef355196461733a385 HTTP/1.1" 200 None 2026-04-21 05:32:44,740 - lava_callback - ERROR - Error processing callback for node 69e468ef355196461733a385: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:59092 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53892 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49204 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39516 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39518 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59276 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59292 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36920 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57568 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45174 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:34:04,789 - lava_callback - INFO - Received callback for node 69e4693b355196461733a3af INFO: 172.20.0.1:49652 - "POST /node/69e4693b355196461733a3af HTTP/1.0" 202 Accepted 2026-04-21 05:34:04,794 - lava_callback - INFO - Starting async job submit for node 69e4693b355196461733a3af 2026-04-21 05:34:04,795 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:34:04,809 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4693b355196461733a3af HTTP/1.1" 200 None 2026-04-21 05:34:04,810 - lava_callback - DEBUG - Processing node 69e4693b355196461733a3af: baseline-arm64-clabbe 2026-04-21 05:34:04,810 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4693b355196461733a3af 2026-04-21 05:34:04,811 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:34:04,824 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4693b355196461733a3af HTTP/1.1" 200 None 2026-04-21 05:34:04,825 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:34:04,852 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:34:04,853 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:34:04,871 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4693b355196461733a3af HTTP/1.1" 200 None 2026-04-21 05:34:04,872 - lava_callback - ERROR - Error processing callback for node 69e4693b355196461733a3af: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49660 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39846 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39860 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41032 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41044 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52170 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42506 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59498 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59766 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40072 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40084 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43366 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:35:24,783 - lava_callback - INFO - Received callback for node 69e46987355196461733a3cd INFO: 172.20.0.1:43376 - "POST /node/69e46987355196461733a3cd HTTP/1.0" 202 Accepted 2026-04-21 05:35:24,788 - lava_callback - INFO - Starting async job submit for node 69e46987355196461733a3cd 2026-04-21 05:35:24,789 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:35:24,811 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46987355196461733a3cd HTTP/1.1" 200 None 2026-04-21 05:35:24,811 - lava_callback - DEBUG - Processing node 69e46987355196461733a3cd: baseline-arm64-clabbe 2026-04-21 05:35:24,812 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e46987355196461733a3cd 2026-04-21 05:35:24,812 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:35:24,825 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46987355196461733a3cd HTTP/1.1" 200 None 2026-04-21 05:35:24,826 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:35:24,838 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:35:24,839 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:35:24,856 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e46987355196461733a3cd HTTP/1.1" 200 None 2026-04-21 05:35:24,857 - lava_callback - ERROR - Error processing callback for node 69e46987355196461733a3cd: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-04-21 05:35:24,944 - lava_callback - INFO - Received callback for node 69e6eef760f5826562c16eb8 INFO: 172.20.0.1:43380 - "POST /node/69e6eef760f5826562c16eb8 HTTP/1.0" 202 Accepted 2026-04-21 05:35:24,951 - lava_callback - INFO - Starting async job submit for node 69e6eef760f5826562c16eb8 2026-04-21 05:35:24,978 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:35:24,998 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6eef760f5826562c16eb8 HTTP/1.1" 200 None 2026-04-21 05:35:24,998 - lava_callback - DEBUG - Processing node 69e6eef760f5826562c16eb8: baseline-nfs-arm64 2026-04-21 05:35:25,555 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 05:35:25,676 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 05:35:25,678 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6eef760f5826562c16eb8/log.txt.gz 2026-04-21 05:35:25,974 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 05:35:26,350 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 05:35:26,352 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6eef760f5826562c16eb8/lava-logs.yaml 2026-04-21 05:35:26,364 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 05:35:26,504 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 05:35:26,506 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69e6eef760f5826562c16eb8/lava_callback.json.gz INFO: 172.20.0.1:43392 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:35:26,533 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:35:26,547 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6eef760f5826562c16eb8 HTTP/1.1" 200 None 2026-04-21 05:35:26,549 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:35:26,562 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 05:35:26,563 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:35:26,581 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6eef760f5826562c16eb8 HTTP/1.1" 200 None 2026-04-21 05:35:26,638 - lava_callback - INFO - Completed processing callback for node 69e6eef760f5826562c16eb8 INFO: 172.20.0.1:46988 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:35:46,142 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:35:46,160 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:41014 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56512 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37416 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41134 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41136 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52972 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:36:24,839 - lava_callback - INFO - Received callback for node 69e469d4355196461733a3f6 INFO: 172.20.0.1:52976 - "POST /node/69e469d4355196461733a3f6 HTTP/1.0" 202 Accepted 2026-04-21 05:36:24,845 - lava_callback - INFO - Starting async job submit for node 69e469d4355196461733a3f6 2026-04-21 05:36:24,846 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:36:24,867 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e469d4355196461733a3f6 HTTP/1.1" 200 None 2026-04-21 05:36:24,868 - lava_callback - DEBUG - Processing node 69e469d4355196461733a3f6: baseline-arm64-clabbe 2026-04-21 05:36:24,868 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e469d4355196461733a3f6 2026-04-21 05:36:24,869 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:36:24,882 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e469d4355196461733a3f6 HTTP/1.1" 200 None 2026-04-21 05:36:24,883 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:36:24,896 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:36:24,897 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:36:24,914 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e469d4355196461733a3f6 HTTP/1.1" 200 None 2026-04-21 05:36:24,915 - lava_callback - ERROR - Error processing callback for node 69e469d4355196461733a3f6: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:52982 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34944 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33298 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35064 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48752 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47278 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47282 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37558 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37566 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51168 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:37:44,902 - lava_callback - INFO - Received callback for node 69e46a20355196461733a40e INFO: 172.20.0.1:48206 - "POST /node/69e46a20355196461733a40e HTTP/1.0" 202 Accepted 2026-04-21 05:37:44,908 - lava_callback - INFO - Starting async job submit for node 69e46a20355196461733a40e 2026-04-21 05:37:44,909 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:37:44,945 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46a20355196461733a40e HTTP/1.1" 200 None 2026-04-21 05:37:44,946 - lava_callback - DEBUG - Processing node 69e46a20355196461733a40e: baseline-arm64-clabbe 2026-04-21 05:37:44,946 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e46a20355196461733a40e 2026-04-21 05:37:44,946 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:37:44,960 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46a20355196461733a40e HTTP/1.1" 200 None 2026-04-21 05:37:44,961 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:37:45,009 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:37:45,011 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:37:45,029 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e46a20355196461733a40e HTTP/1.1" 200 None 2026-04-21 05:37:45,030 - lava_callback - ERROR - Error processing callback for node 69e46a20355196461733a40e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:48218 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60104 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43716 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39824 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39828 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35106 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35114 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49378 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46628 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40946 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:39:04,959 - lava_callback - INFO - Received callback for node 69e46a6c355196461733a434 INFO: 172.20.0.1:60482 - "POST /node/69e46a6c355196461733a434 HTTP/1.0" 202 Accepted 2026-04-21 05:39:04,964 - lava_callback - INFO - Starting async job submit for node 69e46a6c355196461733a434 2026-04-21 05:39:04,965 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:39:04,991 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46a6c355196461733a434 HTTP/1.1" 200 None 2026-04-21 05:39:04,992 - lava_callback - DEBUG - Processing node 69e46a6c355196461733a434: baseline-arm64-clabbe 2026-04-21 05:39:04,992 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e46a6c355196461733a434 2026-04-21 05:39:04,993 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:39:05,006 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46a6c355196461733a434 HTTP/1.1" 200 None 2026-04-21 05:39:05,008 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:39:05,029 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:39:05,032 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:39:05,057 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e46a6c355196461733a434 HTTP/1.1" 200 None 2026-04-21 05:39:05,059 - lava_callback - ERROR - Error processing callback for node 69e46a6c355196461733a434: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:60498 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33244 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33260 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45090 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45098 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47824 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53722 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58644 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39558 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44266 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44268 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40254 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:40:25,029 - lava_callback - INFO - Received callback for node 69e46ab8355196461733a453 INFO: 172.20.0.1:40268 - "POST /node/69e46ab8355196461733a453 HTTP/1.0" 202 Accepted 2026-04-21 05:40:25,034 - lava_callback - INFO - Starting async job submit for node 69e46ab8355196461733a453 2026-04-21 05:40:25,035 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:40:25,049 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46ab8355196461733a453 HTTP/1.1" 200 None 2026-04-21 05:40:25,050 - lava_callback - DEBUG - Processing node 69e46ab8355196461733a453: baseline-arm64-clabbe 2026-04-21 05:40:25,050 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e46ab8355196461733a453 2026-04-21 05:40:25,051 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:40:25,065 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46ab8355196461733a453 HTTP/1.1" 200 None 2026-04-21 05:40:25,067 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:40:25,084 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:40:25,085 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:40:25,112 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e46ab8355196461733a453 HTTP/1.1" 200 None 2026-04-21 05:40:25,113 - lava_callback - ERROR - Error processing callback for node 69e46ab8355196461733a453: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:40280 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38506 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51108 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44802 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51390 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58790 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58794 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47690 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47706 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39672 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:41:45,096 - lava_callback - INFO - Received callback for node 69e46b06355196461733a478 INFO: 172.20.0.1:38170 - "POST /node/69e46b06355196461733a478 HTTP/1.0" 202 Accepted 2026-04-21 05:41:45,101 - lava_callback - INFO - Starting async job submit for node 69e46b06355196461733a478 2026-04-21 05:41:45,102 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:41:45,116 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46b06355196461733a478 HTTP/1.1" 200 None 2026-04-21 05:41:45,117 - lava_callback - DEBUG - Processing node 69e46b06355196461733a478: baseline-arm64-clabbe 2026-04-21 05:41:45,117 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e46b06355196461733a478 2026-04-21 05:41:45,118 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:41:45,130 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46b06355196461733a478 HTTP/1.1" 200 None 2026-04-21 05:41:45,132 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:41:45,144 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:41:45,146 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:41:45,180 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e46b06355196461733a478 HTTP/1.1" 200 None 2026-04-21 05:41:45,182 - lava_callback - ERROR - Error processing callback for node 69e46b06355196461733a478: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:38174 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33266 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40384 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54452 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:54454 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48958 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48974 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60246 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44096 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59664 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:43:05,140 - lava_callback - INFO - Received callback for node 69e46b53355196461733a494 INFO: 172.20.0.1:43546 - "POST /node/69e46b53355196461733a494 HTTP/1.0" 202 Accepted 2026-04-21 05:43:05,146 - lava_callback - INFO - Starting async job submit for node 69e46b53355196461733a494 2026-04-21 05:43:05,147 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:43:05,161 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46b53355196461733a494 HTTP/1.1" 200 None 2026-04-21 05:43:05,162 - lava_callback - DEBUG - Processing node 69e46b53355196461733a494: baseline-arm64-clabbe 2026-04-21 05:43:05,162 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e46b53355196461733a494 2026-04-21 05:43:05,163 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:43:05,176 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46b53355196461733a494 HTTP/1.1" 200 None 2026-04-21 05:43:05,177 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:43:05,190 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:43:05,192 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:43:05,209 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e46b53355196461733a494 HTTP/1.1" 200 None 2026-04-21 05:43:05,210 - lava_callback - ERROR - Error processing callback for node 69e46b53355196461733a494: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:43562 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39296 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39310 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43678 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43690 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51360 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50606 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57550 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:44:05,195 - lava_callback - INFO - Received callback for node 69e46ba0355196461733a4ba INFO: 172.20.0.1:52598 - "POST /node/69e46ba0355196461733a4ba HTTP/1.0" 202 Accepted 2026-04-21 05:44:05,200 - lava_callback - INFO - Starting async job submit for node 69e46ba0355196461733a4ba 2026-04-21 05:44:05,201 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:44:05,216 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46ba0355196461733a4ba HTTP/1.1" 200 None 2026-04-21 05:44:05,216 - lava_callback - DEBUG - Processing node 69e46ba0355196461733a4ba: baseline-arm64-clabbe 2026-04-21 05:44:05,217 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e46ba0355196461733a4ba 2026-04-21 05:44:05,217 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:44:05,231 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46ba0355196461733a4ba HTTP/1.1" 200 None 2026-04-21 05:44:05,232 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:44:05,244 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:44:05,246 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:44:05,266 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e46ba0355196461733a4ba HTTP/1.1" 200 None 2026-04-21 05:44:05,266 - lava_callback - ERROR - Error processing callback for node 69e46ba0355196461733a4ba: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:52606 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52758 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52770 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49162 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49164 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43228 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49620 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35026 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48842 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33914 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33930 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33684 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:45:25,251 - lava_callback - INFO - Received callback for node 69e46bef355196461733a4e0 INFO: 172.20.0.1:33698 - "POST /node/69e46bef355196461733a4e0 HTTP/1.0" 202 Accepted 2026-04-21 05:45:25,257 - lava_callback - INFO - Starting async job submit for node 69e46bef355196461733a4e0 2026-04-21 05:45:25,258 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:45:25,273 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46bef355196461733a4e0 HTTP/1.1" 200 None 2026-04-21 05:45:25,274 - lava_callback - DEBUG - Processing node 69e46bef355196461733a4e0: baseline-arm64-clabbe 2026-04-21 05:45:25,274 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e46bef355196461733a4e0 2026-04-21 05:45:25,275 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:45:25,288 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46bef355196461733a4e0 HTTP/1.1" 200 None 2026-04-21 05:45:25,290 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:45:25,304 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:45:25,306 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:45:25,324 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e46bef355196461733a4e0 HTTP/1.1" 200 None 2026-04-21 05:45:25,326 - lava_callback - ERROR - Error processing callback for node 69e46bef355196461733a4e0: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:33702 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57002 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57634 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53308 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52380 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34304 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34306 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39262 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39276 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57350 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:46:45,305 - lava_callback - INFO - Received callback for node 69e46c3d355196461733a4fa INFO: 172.20.0.1:60206 - "POST /node/69e46c3d355196461733a4fa HTTP/1.0" 202 Accepted 2026-04-21 05:46:45,311 - lava_callback - INFO - Starting async job submit for node 69e46c3d355196461733a4fa 2026-04-21 05:46:45,312 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:46:45,332 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46c3d355196461733a4fa HTTP/1.1" 200 None 2026-04-21 05:46:45,333 - lava_callback - DEBUG - Processing node 69e46c3d355196461733a4fa: baseline-arm64-clabbe 2026-04-21 05:46:45,334 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e46c3d355196461733a4fa 2026-04-21 05:46:45,334 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:46:45,347 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46c3d355196461733a4fa HTTP/1.1" 200 None 2026-04-21 05:46:45,348 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:46:45,361 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:46:45,363 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:46:45,380 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e46c3d355196461733a4fa HTTP/1.1" 200 None 2026-04-21 05:46:45,382 - lava_callback - ERROR - Error processing callback for node 69e46c3d355196461733a4fa: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:60222 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50964 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44394 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38858 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:38862 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54072 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54088 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43880 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38062 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50040 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:48:05,433 - lava_callback - INFO - Received callback for node 69e46c88355196461733a520 INFO: 172.20.0.1:50476 - "POST /node/69e46c88355196461733a520 HTTP/1.0" 202 Accepted 2026-04-21 05:48:05,439 - lava_callback - INFO - Starting async job submit for node 69e46c88355196461733a520 2026-04-21 05:48:05,439 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:48:05,454 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46c88355196461733a520 HTTP/1.1" 200 None 2026-04-21 05:48:05,455 - lava_callback - DEBUG - Processing node 69e46c88355196461733a520: baseline-arm64-clabbe 2026-04-21 05:48:05,455 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e46c88355196461733a520 2026-04-21 05:48:05,456 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:48:05,470 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46c88355196461733a520 HTTP/1.1" 200 None 2026-04-21 05:48:05,472 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:48:05,488 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bca2eed175ae971a59a HTTP/1.1" 200 None 2026-04-21 05:48:05,490 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:48:05,514 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e46c88355196461733a520 HTTP/1.1" 200 None 2026-04-21 05:48:05,516 - lava_callback - ERROR - Error processing callback for node 69e46c88355196461733a520: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:50482 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48958 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48972 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50732 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50744 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46796 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36948 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43916 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47728 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32862 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:32864 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36214 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36222 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38880 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48558 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59102 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50964 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36036 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:36050 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35204 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35210 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55610 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41312 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57946 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33010 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40088 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40090 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52642 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52658 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49928 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54880 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49608 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52326 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55798 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:55806 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59872 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59878 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60574 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53924 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42028 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55662 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40016 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40020 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36424 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36432 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39484 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56884 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60528 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46482 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45420 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45430 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44656 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44668 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48654 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38722 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53524 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59692 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33900 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33910 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44270 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44284 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42842 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41542 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59980 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52196 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40472 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40482 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53924 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53930 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46164 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34126 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45644 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57656 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48794 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48796 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:57:21,982 - lava_callback - INFO - Received callback for node 69e46eac355196461733a5fc INFO: 172.20.0.1:46742 - "POST /node/69e46eac355196461733a5fc HTTP/1.0" 202 Accepted 2026-04-21 05:57:21,987 - lava_callback - INFO - Starting async job submit for node 69e46eac355196461733a5fc 2026-04-21 05:57:21,989 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:57:22,004 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46eac355196461733a5fc HTTP/1.1" 200 None 2026-04-21 05:57:22,005 - lava_callback - DEBUG - Processing node 69e46eac355196461733a5fc: baseline-arm64-baylibre 2026-04-21 05:57:22,005 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e46eac355196461733a5fc 2026-04-21 05:57:22,006 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:57:22,020 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46eac355196461733a5fc HTTP/1.1" 200 None 2026-04-21 05:57:22,021 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:57:22,034 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e3af097f5347282fbdbc87 HTTP/1.1" 200 None 2026-04-21 05:57:22,036 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:57:22,053 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e46eac355196461733a5fc HTTP/1.1" 200 None 2026-04-21 05:57:22,054 - lava_callback - ERROR - Error processing callback for node 69e46eac355196461733a5fc: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:46756 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46758 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46348 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53314 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57956 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53948 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38016 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:38028 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43172 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:58:25,786 - lava_callback - INFO - Received callback for node 69e46efd355196461733a622 INFO: 172.20.0.1:43186 - "POST /node/69e46efd355196461733a622 HTTP/1.0" 202 Accepted 2026-04-21 05:58:25,792 - lava_callback - INFO - Starting async job submit for node 69e46efd355196461733a622 2026-04-21 05:58:25,793 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:58:25,807 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46efd355196461733a622 HTTP/1.1" 200 None 2026-04-21 05:58:25,808 - lava_callback - DEBUG - Processing node 69e46efd355196461733a622: baseline-arm64-clabbe 2026-04-21 05:58:25,808 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e46efd355196461733a622 2026-04-21 05:58:25,808 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:58:25,822 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46efd355196461733a622 HTTP/1.1" 200 None 2026-04-21 05:58:25,824 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:58:25,839 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e3af097f5347282fbdbc87 HTTP/1.1" 200 None 2026-04-21 05:58:25,841 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:58:25,866 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e46efd355196461733a622 HTTP/1.1" 200 None 2026-04-21 05:58:25,868 - lava_callback - ERROR - Error processing callback for node 69e46efd355196461733a622: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:43190 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48600 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35898 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47950 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33036 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49196 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49198 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44682 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44692 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51312 - "GET / HTTP/1.0" 200 OK 2026-04-21 05:59:45,851 - lava_callback - INFO - Received callback for node 69e46f4a355196461733a63a INFO: 172.20.0.1:48116 - "POST /node/69e46f4a355196461733a63a HTTP/1.0" 202 Accepted 2026-04-21 05:59:45,856 - lava_callback - INFO - Starting async job submit for node 69e46f4a355196461733a63a 2026-04-21 05:59:45,857 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:59:45,871 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46f4a355196461733a63a HTTP/1.1" 200 None 2026-04-21 05:59:45,872 - lava_callback - DEBUG - Processing node 69e46f4a355196461733a63a: baseline-arm64-clabbe 2026-04-21 05:59:45,872 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e46f4a355196461733a63a 2026-04-21 05:59:45,873 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:59:45,887 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46f4a355196461733a63a HTTP/1.1" 200 None 2026-04-21 05:59:45,889 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:59:45,925 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e3af097f5347282fbdbc87 HTTP/1.1" 200 None 2026-04-21 05:59:45,927 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 05:59:45,954 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e46f4a355196461733a63a HTTP/1.1" 200 None 2026-04-21 05:59:45,955 - lava_callback - ERROR - Error processing callback for node 69e46f4a355196461733a63a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:48120 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39244 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54416 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45130 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45144 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45546 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45558 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48944 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41306 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44720 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:01:05,913 - lava_callback - INFO - Received callback for node 69e46f98355196461733a65d INFO: 172.20.0.1:36722 - "POST /node/69e46f98355196461733a65d HTTP/1.0" 202 Accepted 2026-04-21 06:01:05,919 - lava_callback - INFO - Starting async job submit for node 69e46f98355196461733a65d 2026-04-21 06:01:05,920 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:01:05,933 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46f98355196461733a65d HTTP/1.1" 200 None 2026-04-21 06:01:05,934 - lava_callback - DEBUG - Processing node 69e46f98355196461733a65d: baseline-arm64-clabbe 2026-04-21 06:01:05,934 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e46f98355196461733a65d 2026-04-21 06:01:05,935 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:01:05,948 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46f98355196461733a65d HTTP/1.1" 200 None 2026-04-21 06:01:05,950 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:01:05,967 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e3af097f5347282fbdbc87 HTTP/1.1" 200 None 2026-04-21 06:01:05,970 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:01:06,044 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e46f98355196461733a65d HTTP/1.1" 200 None 2026-04-21 06:01:06,046 - lava_callback - ERROR - Error processing callback for node 69e46f98355196461733a65d: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:36724 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52218 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52226 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47536 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47542 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47164 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56414 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47988 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39628 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49578 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49588 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53934 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:02:25,998 - lava_callback - INFO - Received callback for node 69e46fe6355196461733a679 INFO: 172.20.0.1:53948 - "POST /node/69e46fe6355196461733a679 HTTP/1.0" 202 Accepted 2026-04-21 06:02:26,003 - lava_callback - INFO - Starting async job submit for node 69e46fe6355196461733a679 2026-04-21 06:02:26,005 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:02:26,019 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46fe6355196461733a679 HTTP/1.1" 200 None 2026-04-21 06:02:26,020 - lava_callback - DEBUG - Processing node 69e46fe6355196461733a679: baseline-arm64-clabbe 2026-04-21 06:02:26,020 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e46fe6355196461733a679 2026-04-21 06:02:26,020 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:02:26,040 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e46fe6355196461733a679 HTTP/1.1" 200 None 2026-04-21 06:02:26,042 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:02:26,055 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e3af097f5347282fbdbc87 HTTP/1.1" 200 None 2026-04-21 06:02:26,056 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:02:26,076 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e46fe6355196461733a679 HTTP/1.1" 200 None 2026-04-21 06:02:26,078 - lava_callback - ERROR - Error processing callback for node 69e46fe6355196461733a679: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:53950 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57692 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49142 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53404 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51784 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37706 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37714 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58518 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58532 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55012 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:03:46,028 - lava_callback - INFO - Received callback for node 69e47037355196461733a69a INFO: 172.20.0.1:41804 - "POST /node/69e47037355196461733a69a HTTP/1.0" 202 Accepted 2026-04-21 06:03:46,037 - lava_callback - INFO - Starting async job submit for node 69e47037355196461733a69a 2026-04-21 06:03:46,038 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:03:46,055 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47037355196461733a69a HTTP/1.1" 200 None 2026-04-21 06:03:46,056 - lava_callback - DEBUG - Processing node 69e47037355196461733a69a: baseline-arm64-clabbe 2026-04-21 06:03:46,056 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47037355196461733a69a 2026-04-21 06:03:46,057 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:03:46,071 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47037355196461733a69a HTTP/1.1" 200 None 2026-04-21 06:03:46,072 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:03:46,108 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e3af097f5347282fbdbc87 HTTP/1.1" 200 None 2026-04-21 06:03:46,109 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:03:46,127 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47037355196461733a69a HTTP/1.1" 200 None 2026-04-21 06:03:46,128 - lava_callback - ERROR - Error processing callback for node 69e47037355196461733a69a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:41812 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50268 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43724 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58578 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58594 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41484 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41492 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44466 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38294 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41446 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:05:06,085 - lava_callback - INFO - Received callback for node 69e47084355196461733a6be INFO: 172.20.0.1:47944 - "POST /node/69e47084355196461733a6be HTTP/1.0" 202 Accepted 2026-04-21 06:05:06,091 - lava_callback - INFO - Starting async job submit for node 69e47084355196461733a6be 2026-04-21 06:05:06,092 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:05:06,107 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47084355196461733a6be HTTP/1.1" 200 None 2026-04-21 06:05:06,107 - lava_callback - DEBUG - Processing node 69e47084355196461733a6be: baseline-arm64-clabbe 2026-04-21 06:05:06,108 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47084355196461733a6be 2026-04-21 06:05:06,108 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:05:06,121 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47084355196461733a6be HTTP/1.1" 200 None 2026-04-21 06:05:06,122 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:05:06,135 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e3af097f5347282fbdbc87 HTTP/1.1" 200 None 2026-04-21 06:05:06,137 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:05:06,159 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47084355196461733a6be HTTP/1.1" 200 None 2026-04-21 06:05:06,160 - lava_callback - ERROR - Error processing callback for node 69e47084355196461733a6be: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47948 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40962 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40964 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42106 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42108 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51796 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57262 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36312 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56294 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54506 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:54508 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60134 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:06:26,127 - lava_callback - INFO - Received callback for node 69e470d0355196461733a6d6 INFO: 172.20.0.1:60140 - "POST /node/69e470d0355196461733a6d6 HTTP/1.0" 202 Accepted 2026-04-21 06:06:26,133 - lava_callback - INFO - Starting async job submit for node 69e470d0355196461733a6d6 2026-04-21 06:06:26,134 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:06:26,148 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e470d0355196461733a6d6 HTTP/1.1" 200 None 2026-04-21 06:06:26,149 - lava_callback - DEBUG - Processing node 69e470d0355196461733a6d6: baseline-arm64-clabbe 2026-04-21 06:06:26,149 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e470d0355196461733a6d6 2026-04-21 06:06:26,150 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:06:26,163 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e470d0355196461733a6d6 HTTP/1.1" 200 None 2026-04-21 06:06:26,164 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:06:26,183 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e3af097f5347282fbdbc87 HTTP/1.1" 200 None 2026-04-21 06:06:26,185 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:06:26,204 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e470d0355196461733a6d6 HTTP/1.1" 200 None 2026-04-21 06:06:26,205 - lava_callback - ERROR - Error processing callback for node 69e470d0355196461733a6d6: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:60154 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52570 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46932 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45560 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45710 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32914 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:32916 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50638 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50654 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47960 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56560 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51770 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55344 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40848 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40864 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37122 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37136 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57984 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40290 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45572 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47092 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36860 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:36874 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55908 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55920 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53400 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38530 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59948 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53018 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34944 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34948 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52856 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52858 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53652 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55468 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57942 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48996 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47126 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47136 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34908 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34922 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36088 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36914 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33054 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40772 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41556 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41570 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49166 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49172 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48146 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44544 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53550 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34184 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36866 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:36870 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58578 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58584 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51506 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55354 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48158 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59668 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33794 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33808 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48870 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48886 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49236 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55178 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35626 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59458 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39990 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40000 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36754 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36760 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56314 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44464 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54770 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60266 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55108 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:55118 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42734 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42740 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57394 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41852 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53564 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45524 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56094 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:56102 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53974 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53976 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48500 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48116 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38494 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35900 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37670 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37674 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37268 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37276 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43690 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46902 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39838 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49244 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37610 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37618 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50330 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50336 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52224 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59116 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38856 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37284 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45136 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45150 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55630 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55632 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54836 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45822 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58084 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60726 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57804 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57806 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44044 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44050 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40158 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51792 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46300 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50206 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38148 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:38150 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48462 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48474 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56382 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52664 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40546 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33386 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39718 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39734 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56156 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56172 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51466 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57612 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41996 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36052 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36004 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:36016 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52326 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52334 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53714 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47180 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51226 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58464 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49704 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49714 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:25:23,435 - lava_callback - INFO - Received callback for node 69e4753e355196461733a700 INFO: 172.20.0.1:44412 - "POST /node/69e4753e355196461733a700 HTTP/1.0" 202 Accepted 2026-04-21 06:25:23,440 - lava_callback - INFO - Starting async job submit for node 69e4753e355196461733a700 2026-04-21 06:25:23,442 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:25:23,456 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4753e355196461733a700 HTTP/1.1" 200 None 2026-04-21 06:25:23,457 - lava_callback - DEBUG - Processing node 69e4753e355196461733a700: baseline-arm-baylibre 2026-04-21 06:25:23,457 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4753e355196461733a700 2026-04-21 06:25:23,458 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:25:23,476 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4753e355196461733a700 HTTP/1.1" 200 None 2026-04-21 06:25:23,478 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:25:23,492 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 06:25:23,494 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:25:23,513 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4753e355196461733a700 HTTP/1.1" 200 None 2026-04-21 06:25:23,515 - lava_callback - ERROR - Error processing callback for node 69e4753e355196461733a700: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:44426 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44442 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36970 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35184 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34552 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49030 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54162 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:54176 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:26:23,470 - lava_callback - INFO - Received callback for node 69e4757b355196461733a714 INFO: 172.20.0.1:44524 - "POST /node/69e4757b355196461733a714 HTTP/1.0" 202 Accepted 2026-04-21 06:26:23,476 - lava_callback - INFO - Starting async job submit for node 69e4757b355196461733a714 2026-04-21 06:26:23,477 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:26:23,491 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4757b355196461733a714 HTTP/1.1" 200 None 2026-04-21 06:26:23,492 - lava_callback - DEBUG - Processing node 69e4757b355196461733a714: baseline-arm-baylibre 2026-04-21 06:26:23,492 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4757b355196461733a714 2026-04-21 06:26:23,493 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:26:23,506 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4757b355196461733a714 HTTP/1.1" 200 None 2026-04-21 06:26:23,508 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:26:23,521 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 06:26:23,523 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:26:23,545 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4757b355196461733a714 HTTP/1.1" 200 None 2026-04-21 06:26:23,546 - lava_callback - ERROR - Error processing callback for node 69e4757b355196461733a714: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:44530 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44536 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34054 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45508 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33674 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45374 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60398 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60408 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41808 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41812 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46846 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57098 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60278 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54836 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43492 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43498 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:28:23,594 - lava_callback - INFO - Received callback for node 69e475f5355196461733a741 INFO: 172.20.0.1:36450 - "POST /node/69e475f5355196461733a741 HTTP/1.0" 202 Accepted 2026-04-21 06:28:23,599 - lava_callback - INFO - Starting async job submit for node 69e475f5355196461733a741 2026-04-21 06:28:23,601 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:28:23,615 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e475f5355196461733a741 HTTP/1.1" 200 None 2026-04-21 06:28:23,616 - lava_callback - DEBUG - Processing node 69e475f5355196461733a741: baseline-arm-baylibre 2026-04-21 06:28:23,616 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e475f5355196461733a741 2026-04-21 06:28:23,617 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:28:23,629 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e475f5355196461733a741 HTTP/1.1" 200 None 2026-04-21 06:28:23,631 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:28:23,644 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 06:28:23,645 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:28:23,669 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e475f5355196461733a741 HTTP/1.1" 200 None 2026-04-21 06:28:23,670 - lava_callback - ERROR - Error processing callback for node 69e475f5355196461733a741: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:36462 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36478 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47660 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:28:43,606 - lava_callback - INFO - Received callback for node 69e47619355196461733a750 INFO: 172.20.0.1:35772 - "POST /node/69e47619355196461733a750 HTTP/1.0" 202 Accepted 2026-04-21 06:28:43,612 - lava_callback - INFO - Starting async job submit for node 69e47619355196461733a750 2026-04-21 06:28:43,614 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:28:43,632 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47619355196461733a750 HTTP/1.1" 200 None 2026-04-21 06:28:43,634 - lava_callback - DEBUG - Processing node 69e47619355196461733a750: baseline-arm-baylibre 2026-04-21 06:28:43,634 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47619355196461733a750 2026-04-21 06:28:43,635 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:28:43,655 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47619355196461733a750 HTTP/1.1" 200 None 2026-04-21 06:28:43,657 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:28:43,672 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 06:28:43,674 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:28:43,697 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47619355196461733a750 HTTP/1.1" 200 None 2026-04-21 06:28:43,699 - lava_callback - ERROR - Error processing callback for node 69e47619355196461733a750: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:35784 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45750 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50030 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43686 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43698 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58522 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58536 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:29:27,148 - lava_callback - INFO - Received callback for node 69e4763f355196461733a768 INFO: 172.20.0.1:58548 - "POST /node/69e4763f355196461733a768 HTTP/1.0" 202 Accepted 2026-04-21 06:29:27,153 - lava_callback - INFO - Starting async job submit for node 69e4763f355196461733a768 2026-04-21 06:29:27,154 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:29:27,169 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4763f355196461733a768 HTTP/1.1" 200 None 2026-04-21 06:29:27,170 - lava_callback - DEBUG - Processing node 69e4763f355196461733a768: baseline-arm-clabbe 2026-04-21 06:29:27,170 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4763f355196461733a768 2026-04-21 06:29:27,171 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:29:27,190 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4763f355196461733a768 HTTP/1.1" 200 None 2026-04-21 06:29:27,191 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:29:27,204 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 06:29:27,205 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:29:27,222 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4763f355196461733a768 HTTP/1.1" 200 None 2026-04-21 06:29:27,223 - lava_callback - ERROR - Error processing callback for node 69e4763f355196461733a768: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:40780 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57456 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56190 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50310 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:30:07,152 - lava_callback - INFO - Received callback for node 69e47664355196461733a775 INFO: 172.20.0.1:50322 - "POST /node/69e47664355196461733a775 HTTP/1.0" 202 Accepted 2026-04-21 06:30:07,158 - lava_callback - INFO - Starting async job submit for node 69e47664355196461733a775 2026-04-21 06:30:07,159 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:30:07,175 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47664355196461733a775 HTTP/1.1" 200 None 2026-04-21 06:30:07,176 - lava_callback - DEBUG - Processing node 69e47664355196461733a775: baseline-arm-clabbe 2026-04-21 06:30:07,176 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47664355196461733a775 2026-04-21 06:30:07,177 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:30:07,191 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47664355196461733a775 HTTP/1.1" 200 None 2026-04-21 06:30:07,193 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:30:07,212 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 06:30:07,214 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:30:07,234 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47664355196461733a775 HTTP/1.1" 200 None 2026-04-21 06:30:07,236 - lava_callback - ERROR - Error processing callback for node 69e47664355196461733a775: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:60036 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60048 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48984 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48992 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46626 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41710 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:30:47,151 - lava_callback - INFO - Received callback for node 69e47689355196461733a788 INFO: 172.20.0.1:41716 - "POST /node/69e47689355196461733a788 HTTP/1.0" 202 Accepted 2026-04-21 06:30:47,156 - lava_callback - INFO - Starting async job submit for node 69e47689355196461733a788 2026-04-21 06:30:47,158 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:30:47,173 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47689355196461733a788 HTTP/1.1" 200 None 2026-04-21 06:30:47,174 - lava_callback - DEBUG - Processing node 69e47689355196461733a788: baseline-arm-clabbe 2026-04-21 06:30:47,174 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47689355196461733a788 2026-04-21 06:30:47,175 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:30:47,188 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47689355196461733a788 HTTP/1.1" 200 None 2026-04-21 06:30:47,190 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:30:47,202 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 06:30:47,204 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:30:47,223 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47689355196461733a788 HTTP/1.1" 200 None 2026-04-21 06:30:47,224 - lava_callback - ERROR - Error processing callback for node 69e47689355196461733a788: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:57736 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36250 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59868 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59878 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58882 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58886 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:31:27,145 - lava_callback - INFO - Received callback for node 69e476ae355196461733a79e INFO: 172.20.0.1:58902 - "POST /node/69e476ae355196461733a79e HTTP/1.0" 202 Accepted 2026-04-21 06:31:27,150 - lava_callback - INFO - Starting async job submit for node 69e476ae355196461733a79e 2026-04-21 06:31:27,152 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:31:27,165 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e476ae355196461733a79e HTTP/1.1" 200 None 2026-04-21 06:31:27,166 - lava_callback - DEBUG - Processing node 69e476ae355196461733a79e: baseline-arm-clabbe 2026-04-21 06:31:27,167 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e476ae355196461733a79e 2026-04-21 06:31:27,167 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:31:27,181 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e476ae355196461733a79e HTTP/1.1" 200 None 2026-04-21 06:31:27,183 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:31:27,196 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 06:31:27,198 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:31:27,216 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e476ae355196461733a79e HTTP/1.1" 200 None 2026-04-21 06:31:27,217 - lava_callback - ERROR - Error processing callback for node 69e476ae355196461733a79e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:50320 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58890 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43742 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53918 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:32:07,198 - lava_callback - INFO - Received callback for node 69e476d4355196461733a7b4 INFO: 172.20.0.1:53924 - "POST /node/69e476d4355196461733a7b4 HTTP/1.0" 202 Accepted 2026-04-21 06:32:07,203 - lava_callback - INFO - Starting async job submit for node 69e476d4355196461733a7b4 2026-04-21 06:32:07,205 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:32:07,219 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e476d4355196461733a7b4 HTTP/1.1" 200 None 2026-04-21 06:32:07,220 - lava_callback - DEBUG - Processing node 69e476d4355196461733a7b4: baseline-arm-clabbe 2026-04-21 06:32:07,220 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e476d4355196461733a7b4 2026-04-21 06:32:07,221 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:32:07,234 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e476d4355196461733a7b4 HTTP/1.1" 200 None 2026-04-21 06:32:07,236 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:32:07,251 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 06:32:07,252 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:32:07,271 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e476d4355196461733a7b4 HTTP/1.1" 200 None 2026-04-21 06:32:07,273 - lava_callback - ERROR - Error processing callback for node 69e476d4355196461733a7b4: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:39608 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39610 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57798 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57812 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43994 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43702 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:32:47,148 - lava_callback - INFO - Received callback for node 69e476f9355196461733a7c1 INFO: 172.20.0.1:43712 - "POST /node/69e476f9355196461733a7c1 HTTP/1.0" 202 Accepted 2026-04-21 06:32:47,154 - lava_callback - INFO - Starting async job submit for node 69e476f9355196461733a7c1 2026-04-21 06:32:47,155 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:32:47,170 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e476f9355196461733a7c1 HTTP/1.1" 200 None 2026-04-21 06:32:47,171 - lava_callback - DEBUG - Processing node 69e476f9355196461733a7c1: baseline-arm-clabbe 2026-04-21 06:32:47,171 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e476f9355196461733a7c1 2026-04-21 06:32:47,171 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:32:47,185 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e476f9355196461733a7c1 HTTP/1.1" 200 None 2026-04-21 06:32:47,186 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:32:47,199 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 06:32:47,200 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:32:47,219 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e476f9355196461733a7c1 HTTP/1.1" 200 None 2026-04-21 06:32:47,220 - lava_callback - ERROR - Error processing callback for node 69e476f9355196461733a7c1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:43026 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48496 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37302 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37316 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53330 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53340 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:33:27,234 - lava_callback - INFO - Received callback for node 69e47724355196461733a7d8 INFO: 172.20.0.1:53346 - "POST /node/69e47724355196461733a7d8 HTTP/1.0" 202 Accepted 2026-04-21 06:33:27,239 - lava_callback - INFO - Starting async job submit for node 69e47724355196461733a7d8 2026-04-21 06:33:27,240 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:33:27,260 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47724355196461733a7d8 HTTP/1.1" 200 None 2026-04-21 06:33:27,261 - lava_callback - DEBUG - Processing node 69e47724355196461733a7d8: baseline-arm-clabbe 2026-04-21 06:33:27,261 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47724355196461733a7d8 2026-04-21 06:33:27,262 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:33:27,275 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47724355196461733a7d8 HTTP/1.1" 200 None 2026-04-21 06:33:27,277 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:33:27,290 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 06:33:27,291 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:33:27,309 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47724355196461733a7d8 HTTP/1.1" 200 None 2026-04-21 06:33:27,310 - lava_callback - ERROR - Error processing callback for node 69e47724355196461733a7d8: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:46758 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51292 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33558 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38332 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48582 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48592 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40990 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41002 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:34:27,301 - lava_callback - INFO - Received callback for node 69e47760355196461733a7ed INFO: 172.20.0.1:41008 - "POST /node/69e47760355196461733a7ed HTTP/1.0" 202 Accepted 2026-04-21 06:34:27,307 - lava_callback - INFO - Starting async job submit for node 69e47760355196461733a7ed 2026-04-21 06:34:27,308 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:34:27,323 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47760355196461733a7ed HTTP/1.1" 200 None 2026-04-21 06:34:27,323 - lava_callback - DEBUG - Processing node 69e47760355196461733a7ed: baseline-arm-clabbe 2026-04-21 06:34:27,324 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47760355196461733a7ed 2026-04-21 06:34:27,324 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:34:27,345 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47760355196461733a7ed HTTP/1.1" 200 None 2026-04-21 06:34:27,346 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:34:27,360 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 06:34:27,362 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:34:27,380 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47760355196461733a7ed HTTP/1.1" 200 None 2026-04-21 06:34:27,381 - lava_callback - ERROR - Error processing callback for node 69e47760355196461733a7ed: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:43734 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60338 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57888 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37286 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56934 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:56950 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54374 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54380 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:35:27,392 - lava_callback - INFO - Received callback for node 69e4779c355196461733a800 INFO: 172.20.0.1:54392 - "POST /node/69e4779c355196461733a800 HTTP/1.0" 202 Accepted 2026-04-21 06:35:27,398 - lava_callback - INFO - Starting async job submit for node 69e4779c355196461733a800 2026-04-21 06:35:27,399 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:35:27,412 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4779c355196461733a800 HTTP/1.1" 200 None 2026-04-21 06:35:27,413 - lava_callback - DEBUG - Processing node 69e4779c355196461733a800: baseline-arm-clabbe 2026-04-21 06:35:27,414 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e4779c355196461733a800 2026-04-21 06:35:27,414 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:35:27,427 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e4779c355196461733a800 HTTP/1.1" 200 None 2026-04-21 06:35:27,428 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:35:27,440 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e10bb42eed175ae971a585 HTTP/1.1" 200 None 2026-04-21 06:35:27,442 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:35:27,459 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e4779c355196461733a800 HTTP/1.1" 200 None 2026-04-21 06:35:27,461 - lava_callback - ERROR - Error processing callback for node 69e4779c355196461733a800: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-04-21 06:35:35,271 - lava_callback - INFO - Received callback for node 69e678bedc041e577b77b2f1 INFO: 172.20.0.1:59604 - "POST /node/69e678bedc041e577b77b2f1 HTTP/1.0" 202 Accepted 2026-04-21 06:35:35,277 - lava_callback - INFO - Starting async job submit for node 69e678bedc041e577b77b2f1 2026-04-21 06:35:35,314 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:35:35,328 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e678bedc041e577b77b2f1 HTTP/1.1" 200 None 2026-04-21 06:35:35,329 - lava_callback - DEBUG - Processing node 69e678bedc041e577b77b2f1: baseline-arm64-broonie 2026-04-21 06:35:35,546 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 06:35:35,695 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 06:35:35,696 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e678bedc041e577b77b2f1/log.txt.gz 2026-04-21 06:35:35,819 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 06:35:36,002 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 06:35:36,004 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e678bedc041e577b77b2f1/lava-logs.yaml 2026-04-21 06:35:36,009 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 06:35:36,257 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 06:35:36,258 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e678bedc041e577b77b2f1/lava_callback.json.gz 2026-04-21 06:35:36,295 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:35:36,310 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e678bedc041e577b77b2f1 HTTP/1.1" 200 None 2026-04-21 06:35:36,312 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:35:36,325 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e66a77dc041e577b7785d5 HTTP/1.1" 200 None 2026-04-21 06:35:36,327 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:35:36,346 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e678bedc041e577b77b2f1 HTTP/1.1" 200 None 2026-04-21 06:35:36,416 - lava_callback - INFO - Completed processing callback for node 69e678bedc041e577b77b2f1 INFO: 172.20.0.1:59606 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:35:46,173 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:35:46,201 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:33064 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52226 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40326 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41308 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41314 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35496 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35500 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57268 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36892 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34864 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53708 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37356 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37366 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39300 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39302 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33370 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37848 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51440 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60778 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59698 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59710 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45958 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45974 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38710 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44962 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46536 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56908 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54892 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:54894 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52344 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52354 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43554 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56202 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48706 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54874 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36220 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:36228 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48458 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48472 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38680 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51516 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:40:55,151 - lava_callback - INFO - Received callback for node 69e69240dc041e577b77c113 INFO: 172.20.0.1:38594 - "POST /node/69e69240dc041e577b77c113 HTTP/1.0" 202 Accepted 2026-04-21 06:40:55,157 - lava_callback - INFO - Starting async job submit for node 69e69240dc041e577b77c113 2026-04-21 06:40:55,214 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:40:55,229 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e69240dc041e577b77c113 HTTP/1.1" 200 None 2026-04-21 06:40:55,229 - lava_callback - DEBUG - Processing node 69e69240dc041e577b77c113: baseline-arm64-broonie 2026-04-21 06:40:55,440 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 06:40:55,549 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 06:40:55,550 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e69240dc041e577b77c113/log.txt.gz 2026-04-21 06:40:55,670 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 06:40:55,827 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 06:40:55,829 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e69240dc041e577b77c113/lava-logs.yaml 2026-04-21 06:40:55,835 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 06:40:55,925 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 06:40:55,926 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e69240dc041e577b77c113/lava_callback.json.gz 2026-04-21 06:40:55,962 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:40:55,975 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e69240dc041e577b77c113 HTTP/1.1" 200 None 2026-04-21 06:40:55,977 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:40:55,990 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6522047863bb428fe4af5 HTTP/1.1" 200 None 2026-04-21 06:40:55,991 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:40:56,008 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e69240dc041e577b77c113 HTTP/1.1" 200 None 2026-04-21 06:40:56,078 - lava_callback - INFO - Completed processing callback for node 69e69240dc041e577b77c113 INFO: 172.20.0.1:38596 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53428 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59452 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 06:41:16,204 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:41:16,228 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:59456 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36652 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36668 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58474 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58832 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41672 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56474 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41192 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41206 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53830 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53840 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57082 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32966 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33470 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48620 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58908 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58920 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60846 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60862 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:43:35,268 - lava_callback - INFO - Received callback for node 69e6cb9960f5826562c15c6a INFO: 172.20.0.1:43980 - "POST /node/69e6cb9960f5826562c15c6a HTTP/1.0" 202 Accepted 2026-04-21 06:43:35,274 - lava_callback - INFO - Starting async job submit for node 69e6cb9960f5826562c15c6a 2026-04-21 06:43:35,310 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:43:35,325 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb9960f5826562c15c6a HTTP/1.1" 200 None 2026-04-21 06:43:35,325 - lava_callback - DEBUG - Processing node 69e6cb9960f5826562c15c6a: baseline-arm64-broonie 2026-04-21 06:43:35,533 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 06:43:35,633 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 06:43:35,635 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cb9960f5826562c15c6a/log.txt.gz 2026-04-21 06:43:35,763 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 06:43:35,922 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 06:43:35,924 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cb9960f5826562c15c6a/lava-logs.yaml 2026-04-21 06:43:35,929 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 06:43:36,027 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 06:43:36,028 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6cb9960f5826562c15c6a/lava_callback.json.gz 2026-04-21 06:43:36,073 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:43:36,087 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6cb9960f5826562c15c6a HTTP/1.1" 200 None 2026-04-21 06:43:36,088 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:43:36,102 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 06:43:36,103 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:43:36,126 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6cb9960f5826562c15c6a HTTP/1.1" 200 None 2026-04-21 06:43:36,211 - lava_callback - INFO - Completed processing callback for node 69e6cb9960f5826562c15c6a INFO: 172.20.0.1:43996 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:43:46,235 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:43:46,250 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:48154 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39724 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46852 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43396 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43398 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39072 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39082 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33718 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52096 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38920 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43366 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49378 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49380 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:45:24,604 - lava_callback - INFO - Received callback for node 69e479f1355196461733a842 INFO: 172.20.0.1:47274 - "POST /node/69e479f1355196461733a842 HTTP/1.0" 202 Accepted 2026-04-21 06:45:24,609 - lava_callback - INFO - Starting async job submit for node 69e479f1355196461733a842 2026-04-21 06:45:24,610 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:45:24,625 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e479f1355196461733a842 HTTP/1.1" 200 None 2026-04-21 06:45:24,626 - lava_callback - DEBUG - Processing node 69e479f1355196461733a842: baseline-arm-baylibre 2026-04-21 06:45:24,626 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e479f1355196461733a842 2026-04-21 06:45:24,627 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:45:24,640 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e479f1355196461733a842 HTTP/1.1" 200 None 2026-04-21 06:45:24,642 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:45:24,655 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 06:45:24,657 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:45:24,679 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e479f1355196461733a842 HTTP/1.1" 200 None 2026-04-21 06:45:24,680 - lava_callback - ERROR - Error processing callback for node 69e479f1355196461733a842: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47284 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47286 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36154 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:45:44,473 - lava_callback - INFO - Received callback for node 69e47a16355196461733a84b INFO: 172.20.0.1:37648 - "POST /node/69e47a16355196461733a84b HTTP/1.0" 202 Accepted 2026-04-21 06:45:44,479 - lava_callback - INFO - Starting async job submit for node 69e47a16355196461733a84b 2026-04-21 06:45:44,480 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:45:44,493 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47a16355196461733a84b HTTP/1.1" 200 None 2026-04-21 06:45:44,494 - lava_callback - DEBUG - Processing node 69e47a16355196461733a84b: baseline-arm-baylibre 2026-04-21 06:45:44,495 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47a16355196461733a84b 2026-04-21 06:45:44,495 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:45:44,508 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47a16355196461733a84b HTTP/1.1" 200 None 2026-04-21 06:45:44,510 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:45:44,522 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 06:45:44,524 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:45:45,568 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47a16355196461733a84b HTTP/1.1" 200 None 2026-04-21 06:45:45,569 - lava_callback - ERROR - Error processing callback for node 69e47a16355196461733a84b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:37650 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41424 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34442 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33634 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33636 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:46:24,492 - lava_callback - INFO - Received callback for node 69e47a3b355196461733a85f INFO: 172.20.0.1:36244 - "POST /node/69e47a3b355196461733a85f HTTP/1.0" 202 Accepted 2026-04-21 06:46:24,498 - lava_callback - INFO - Starting async job submit for node 69e47a3b355196461733a85f 2026-04-21 06:46:24,498 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:46:24,513 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47a3b355196461733a85f HTTP/1.1" 200 None 2026-04-21 06:46:24,514 - lava_callback - DEBUG - Processing node 69e47a3b355196461733a85f: baseline-arm-baylibre 2026-04-21 06:46:24,514 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47a3b355196461733a85f 2026-04-21 06:46:24,515 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:46:24,528 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47a3b355196461733a85f HTTP/1.1" 200 None 2026-04-21 06:46:24,530 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:46:24,543 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 06:46:24,544 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:46:24,563 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47a3b355196461733a85f HTTP/1.1" 200 None 2026-04-21 06:46:24,564 - lava_callback - ERROR - Error processing callback for node 69e47a3b355196461733a85f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:36246 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36258 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37684 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60676 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57298 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:47:04,622 - lava_callback - INFO - Received callback for node 69e47a5f355196461733a86b INFO: 172.20.0.1:41640 - "POST /node/69e47a5f355196461733a86b HTTP/1.0" 202 Accepted 2026-04-21 06:47:04,631 - lava_callback - INFO - Starting async job submit for node 69e47a5f355196461733a86b 2026-04-21 06:47:04,633 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:47:04,649 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47a5f355196461733a86b HTTP/1.1" 200 None 2026-04-21 06:47:04,650 - lava_callback - DEBUG - Processing node 69e47a5f355196461733a86b: baseline-arm-baylibre 2026-04-21 06:47:04,651 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47a5f355196461733a86b 2026-04-21 06:47:04,651 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:47:04,667 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47a5f355196461733a86b HTTP/1.1" 200 None 2026-04-21 06:47:04,669 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:47:04,683 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 06:47:04,685 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:47:04,723 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47a5f355196461733a86b HTTP/1.1" 200 None 2026-04-21 06:47:04,725 - lava_callback - ERROR - Error processing callback for node 69e47a5f355196461733a86b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:41642 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39636 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39642 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49384 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49386 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60276 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:47:44,555 - lava_callback - INFO - Received callback for node 69e47a85355196461733a87b INFO: 172.20.0.1:43482 - "POST /node/69e47a85355196461733a87b HTTP/1.0" 202 Accepted 2026-04-21 06:47:44,561 - lava_callback - INFO - Starting async job submit for node 69e47a85355196461733a87b 2026-04-21 06:47:44,562 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:47:44,575 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47a85355196461733a87b HTTP/1.1" 200 None 2026-04-21 06:47:44,576 - lava_callback - DEBUG - Processing node 69e47a85355196461733a87b: baseline-arm-baylibre 2026-04-21 06:47:44,577 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47a85355196461733a87b 2026-04-21 06:47:44,577 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:47:44,590 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47a85355196461733a87b HTTP/1.1" 200 None 2026-04-21 06:47:44,591 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:47:44,603 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 06:47:44,605 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:47:44,623 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47a85355196461733a87b HTTP/1.1" 200 None 2026-04-21 06:47:44,624 - lava_callback - ERROR - Error processing callback for node 69e47a85355196461733a87b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:43492 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56782 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36792 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40256 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40262 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:48:24,663 - lava_callback - INFO - Received callback for node 69e47aaa355196461733a892 INFO: 172.20.0.1:41582 - "POST /node/69e47aaa355196461733a892 HTTP/1.0" 202 Accepted 2026-04-21 06:48:24,669 - lava_callback - INFO - Starting async job submit for node 69e47aaa355196461733a892 2026-04-21 06:48:24,670 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:48:24,684 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47aaa355196461733a892 HTTP/1.1" 200 None 2026-04-21 06:48:24,685 - lava_callback - DEBUG - Processing node 69e47aaa355196461733a892: baseline-arm-baylibre 2026-04-21 06:48:24,686 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47aaa355196461733a892 2026-04-21 06:48:24,686 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:48:24,700 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47aaa355196461733a892 HTTP/1.1" 200 None 2026-04-21 06:48:24,702 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:48:24,715 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 06:48:24,717 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:48:24,736 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47aaa355196461733a892 HTTP/1.1" 200 None 2026-04-21 06:48:24,737 - lava_callback - ERROR - Error processing callback for node 69e47aaa355196461733a892: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:41592 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41606 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57768 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50378 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:48:55,497 - lava_callback - INFO - Received callback for node 69e6d52360f5826562c16263 INFO: 172.20.0.1:33250 - "POST /node/69e6d52360f5826562c16263 HTTP/1.0" 202 Accepted 2026-04-21 06:48:55,504 - lava_callback - INFO - Starting async job submit for node 69e6d52360f5826562c16263 2026-04-21 06:48:55,544 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:48:55,558 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6d52360f5826562c16263 HTTP/1.1" 200 None 2026-04-21 06:48:55,559 - lava_callback - DEBUG - Processing node 69e6d52360f5826562c16263: baseline-arm64-broonie 2026-04-21 06:48:55,790 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 06:48:56,056 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 06:48:56,058 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6d52360f5826562c16263/log.txt.gz 2026-04-21 06:48:56,179 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 06:48:56,335 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 06:48:56,337 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6d52360f5826562c16263/lava-logs.yaml 2026-04-21 06:48:56,342 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 06:48:56,468 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 06:48:56,470 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6d52360f5826562c16263/lava_callback.json.gz INFO: 172.20.0.1:33258 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:48:56,540 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:48:56,564 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6d52360f5826562c16263 HTTP/1.1" 200 None 2026-04-21 06:48:56,565 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:48:56,580 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e66a77dc041e577b7785d5 HTTP/1.1" 200 None 2026-04-21 06:48:56,582 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:48:56,600 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6d52360f5826562c16263 HTTP/1.1" 200 None 2026-04-21 06:48:56,674 - lava_callback - INFO - Completed processing callback for node 69e6d52360f5826562c16263 INFO: 172.20.0.1:33740 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:49:07,948 - lava_callback - INFO - Received callback for node 69e47acf355196461733a8a2 INFO: 172.20.0.1:33756 - "POST /node/69e47acf355196461733a8a2 HTTP/1.0" 202 Accepted 2026-04-21 06:49:07,953 - lava_callback - INFO - Starting async job submit for node 69e47acf355196461733a8a2 2026-04-21 06:49:07,954 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:49:07,974 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47acf355196461733a8a2 HTTP/1.1" 200 None 2026-04-21 06:49:07,975 - lava_callback - DEBUG - Processing node 69e47acf355196461733a8a2: baseline-arm-clabbe 2026-04-21 06:49:07,975 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47acf355196461733a8a2 2026-04-21 06:49:07,976 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:49:07,988 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47acf355196461733a8a2 HTTP/1.1" 200 None 2026-04-21 06:49:07,990 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:49:08,003 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 06:49:08,004 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:49:08,021 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47acf355196461733a8a2 HTTP/1.1" 200 None 2026-04-21 06:49:08,022 - lava_callback - ERROR - Error processing callback for node 69e47acf355196461733a8a2: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54004 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 06:49:16,254 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:49:16,269 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:54008 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:49:24,652 - lava_callback - INFO - Received callback for node 69e47aee355196461733a8b1 INFO: 172.20.0.1:51454 - "POST /node/69e47aee355196461733a8b1 HTTP/1.0" 202 Accepted 2026-04-21 06:49:24,657 - lava_callback - INFO - Starting async job submit for node 69e47aee355196461733a8b1 2026-04-21 06:49:24,658 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:49:24,695 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47aee355196461733a8b1 HTTP/1.1" 200 None 2026-04-21 06:49:24,696 - lava_callback - DEBUG - Processing node 69e47aee355196461733a8b1: baseline-arm-baylibre 2026-04-21 06:49:24,697 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47aee355196461733a8b1 2026-04-21 06:49:24,698 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:49:24,717 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47aee355196461733a8b1 HTTP/1.1" 200 None 2026-04-21 06:49:24,720 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:49:24,734 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 06:49:24,736 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:49:24,753 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47aee355196461733a8b1 HTTP/1.1" 200 None 2026-04-21 06:49:24,754 - lava_callback - ERROR - Error processing callback for node 69e47aee355196461733a8b1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:51460 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51474 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59294 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41012 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53262 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47490 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:50:07,954 - lava_callback - INFO - Received callback for node 69e47b17355196461733a8c0 INFO: 172.20.0.1:47502 - "POST /node/69e47b17355196461733a8c0 HTTP/1.0" 202 Accepted 2026-04-21 06:50:07,960 - lava_callback - INFO - Starting async job submit for node 69e47b17355196461733a8c0 2026-04-21 06:50:07,961 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:50:07,976 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47b17355196461733a8c0 HTTP/1.1" 200 None 2026-04-21 06:50:07,977 - lava_callback - DEBUG - Processing node 69e47b17355196461733a8c0: baseline-arm-clabbe 2026-04-21 06:50:07,977 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47b17355196461733a8c0 2026-04-21 06:50:07,978 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:50:07,991 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47b17355196461733a8c0 HTTP/1.1" 200 None 2026-04-21 06:50:07,992 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:50:08,007 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 06:50:08,009 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:50:08,038 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47b17355196461733a8c0 HTTP/1.1" 200 None 2026-04-21 06:50:08,039 - lava_callback - ERROR - Error processing callback for node 69e47b17355196461733a8c0: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:56614 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:56624 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43300 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43316 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40096 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50176 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:50:48,034 - lava_callback - INFO - Received callback for node 69e47b3b355196461733a8d1 INFO: 172.20.0.1:50184 - "POST /node/69e47b3b355196461733a8d1 HTTP/1.0" 202 Accepted 2026-04-21 06:50:48,039 - lava_callback - INFO - Starting async job submit for node 69e47b3b355196461733a8d1 2026-04-21 06:50:48,040 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:50:48,055 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47b3b355196461733a8d1 HTTP/1.1" 200 None 2026-04-21 06:50:48,056 - lava_callback - DEBUG - Processing node 69e47b3b355196461733a8d1: baseline-arm-clabbe 2026-04-21 06:50:48,056 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47b3b355196461733a8d1 2026-04-21 06:50:48,057 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:50:48,071 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47b3b355196461733a8d1 HTTP/1.1" 200 None 2026-04-21 06:50:48,072 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:50:48,086 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 06:50:48,088 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:50:48,121 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47b3b355196461733a8d1 HTTP/1.1" 200 None 2026-04-21 06:50:48,123 - lava_callback - ERROR - Error processing callback for node 69e47b3b355196461733a8d1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:48608 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48900 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35692 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35702 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:51:24,746 - lava_callback - INFO - Received callback for node 69e47b5f355196461733a8e2 INFO: 172.20.0.1:49226 - "POST /node/69e47b5f355196461733a8e2 HTTP/1.0" 202 Accepted 2026-04-21 06:51:24,752 - lava_callback - INFO - Starting async job submit for node 69e47b5f355196461733a8e2 2026-04-21 06:51:24,753 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:51:24,769 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47b5f355196461733a8e2 HTTP/1.1" 200 None 2026-04-21 06:51:24,770 - lava_callback - DEBUG - Processing node 69e47b5f355196461733a8e2: baseline-arm-baylibre 2026-04-21 06:51:24,771 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47b5f355196461733a8e2 2026-04-21 06:51:24,771 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:51:24,786 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47b5f355196461733a8e2 HTTP/1.1" 200 None 2026-04-21 06:51:24,787 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:51:24,802 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 06:51:24,804 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:51:24,823 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47b5f355196461733a8e2 HTTP/1.1" 200 None 2026-04-21 06:51:24,825 - lava_callback - ERROR - Error processing callback for node 69e47b5f355196461733a8e2: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49238 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49244 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46544 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:51:37,274 - lava_callback - INFO - Received callback for node 69e6ebb860f5826562c16d4a INFO: 172.20.0.1:46546 - "POST /node/69e6ebb860f5826562c16d4a HTTP/1.0" 202 Accepted 2026-04-21 06:51:37,280 - lava_callback - INFO - Starting async job submit for node 69e6ebb860f5826562c16d4a 2026-04-21 06:51:37,315 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:51:37,329 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6ebb860f5826562c16d4a HTTP/1.1" 200 None 2026-04-21 06:51:37,330 - lava_callback - DEBUG - Processing node 69e6ebb860f5826562c16d4a: baseline-arm64-broonie 2026-04-21 06:51:37,560 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 06:51:37,835 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 06:51:37,836 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6ebb860f5826562c16d4a/log.txt.gz 2026-04-21 06:51:37,960 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 06:51:38,292 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 06:51:38,294 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6ebb860f5826562c16d4a/lava-logs.yaml 2026-04-21 06:51:38,299 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 06:51:38,407 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 06:51:38,409 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e6ebb860f5826562c16d4a/lava_callback.json.gz 2026-04-21 06:51:38,446 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:51:38,460 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6ebb860f5826562c16d4a HTTP/1.1" 200 None 2026-04-21 06:51:38,462 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:51:38,476 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6522047863bb428fe4af5 HTTP/1.1" 200 None 2026-04-21 06:51:38,477 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:51:38,495 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e6ebb860f5826562c16d4a HTTP/1.1" 200 None 2026-04-21 06:51:38,572 - lava_callback - INFO - Completed processing callback for node 69e6ebb860f5826562c16d4a 2026-04-21 06:51:46,272 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:51:46,288 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:39070 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54096 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45262 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:52:08,117 - lava_callback - INFO - Received callback for node 69e47b87355196461733a8f2 INFO: 172.20.0.1:45274 - "POST /node/69e47b87355196461733a8f2 HTTP/1.0" 202 Accepted 2026-04-21 06:52:08,123 - lava_callback - INFO - Starting async job submit for node 69e47b87355196461733a8f2 2026-04-21 06:52:08,124 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:52:08,138 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47b87355196461733a8f2 HTTP/1.1" 200 None 2026-04-21 06:52:08,140 - lava_callback - DEBUG - Processing node 69e47b87355196461733a8f2: baseline-arm-clabbe 2026-04-21 06:52:08,140 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47b87355196461733a8f2 2026-04-21 06:52:08,140 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:52:08,154 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47b87355196461733a8f2 HTTP/1.1" 200 None 2026-04-21 06:52:08,156 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:52:08,173 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 06:52:08,174 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:52:08,193 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47b87355196461733a8f2 HTTP/1.1" 200 None 2026-04-21 06:52:08,195 - lava_callback - ERROR - Error processing callback for node 69e47b87355196461733a8f2: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:39888 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39890 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60610 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60622 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57430 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:52:44,865 - lava_callback - INFO - Received callback for node 69e47ba9355196461733a8fe INFO: 172.20.0.1:36784 - "POST /node/69e47ba9355196461733a8fe HTTP/1.0" 202 Accepted 2026-04-21 06:52:44,871 - lava_callback - INFO - Starting async job submit for node 69e47ba9355196461733a8fe 2026-04-21 06:52:44,872 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:52:44,887 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47ba9355196461733a8fe HTTP/1.1" 200 None 2026-04-21 06:52:44,888 - lava_callback - DEBUG - Processing node 69e47ba9355196461733a8fe: baseline-arm-baylibre 2026-04-21 06:52:44,888 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47ba9355196461733a8fe 2026-04-21 06:52:44,889 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:52:44,903 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47ba9355196461733a8fe HTTP/1.1" 200 None 2026-04-21 06:52:44,905 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:52:44,920 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 06:52:44,921 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:52:44,940 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47ba9355196461733a8fe HTTP/1.1" 200 None 2026-04-21 06:52:44,942 - lava_callback - ERROR - Error processing callback for node 69e47ba9355196461733a8fe: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:36794 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39154 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50506 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32846 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:32860 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34452 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34466 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:53:28,212 - lava_callback - INFO - Received callback for node 69e47bd1355196461733a912 INFO: 172.20.0.1:34482 - "POST /node/69e47bd1355196461733a912 HTTP/1.0" 202 Accepted 2026-04-21 06:53:28,218 - lava_callback - INFO - Starting async job submit for node 69e47bd1355196461733a912 2026-04-21 06:53:28,220 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:53:28,237 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47bd1355196461733a912 HTTP/1.1" 200 None 2026-04-21 06:53:28,238 - lava_callback - DEBUG - Processing node 69e47bd1355196461733a912: baseline-arm-clabbe 2026-04-21 06:53:28,238 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47bd1355196461733a912 2026-04-21 06:53:28,239 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:53:28,253 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47bd1355196461733a912 HTTP/1.1" 200 None 2026-04-21 06:53:28,254 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:53:28,268 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 06:53:28,270 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:53:28,289 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47bd1355196461733a912 HTTP/1.1" 200 None 2026-04-21 06:53:28,290 - lava_callback - ERROR - Error processing callback for node 69e47bd1355196461733a912: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:58100 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56596 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:53:48,164 - lava_callback - INFO - Received callback for node 69e47bf7355196461733a91e INFO: 172.20.0.1:56604 - "POST /node/69e47bf7355196461733a91e HTTP/1.0" 202 Accepted 2026-04-21 06:53:48,169 - lava_callback - INFO - Starting async job submit for node 69e47bf7355196461733a91e 2026-04-21 06:53:48,170 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:53:48,184 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47bf7355196461733a91e HTTP/1.1" 200 None 2026-04-21 06:53:48,185 - lava_callback - DEBUG - Processing node 69e47bf7355196461733a91e: baseline-arm-clabbe 2026-04-21 06:53:48,186 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47bf7355196461733a91e 2026-04-21 06:53:48,186 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:53:48,199 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47bf7355196461733a91e HTTP/1.1" 200 None 2026-04-21 06:53:48,200 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:53:48,213 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 06:53:48,214 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:53:48,231 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47bf7355196461733a91e HTTP/1.1" 200 None 2026-04-21 06:53:48,233 - lava_callback - ERROR - Error processing callback for node 69e47bf7355196461733a91e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:58884 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38154 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60314 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60326 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:54:24,968 - lava_callback - INFO - Received callback for node 69e47c19355196461733a92e INFO: 172.20.0.1:57146 - "POST /node/69e47c19355196461733a92e HTTP/1.0" 202 Accepted 2026-04-21 06:54:24,974 - lava_callback - INFO - Starting async job submit for node 69e47c19355196461733a92e 2026-04-21 06:54:24,976 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:54:25,004 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47c19355196461733a92e HTTP/1.1" 200 None 2026-04-21 06:54:25,005 - lava_callback - DEBUG - Processing node 69e47c19355196461733a92e: baseline-arm-baylibre 2026-04-21 06:54:25,005 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47c19355196461733a92e 2026-04-21 06:54:25,006 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:54:25,022 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47c19355196461733a92e HTTP/1.1" 200 None 2026-04-21 06:54:25,023 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:54:25,037 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 06:54:25,038 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:54:25,061 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47c19355196461733a92e HTTP/1.1" 200 None 2026-04-21 06:54:25,063 - lava_callback - ERROR - Error processing callback for node 69e47c19355196461733a92e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:57150 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57152 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46406 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39908 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45230 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40578 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:55:08,190 - lava_callback - INFO - Received callback for node 69e47c41355196461733a939 INFO: 172.20.0.1:40580 - "POST /node/69e47c41355196461733a939 HTTP/1.0" 202 Accepted 2026-04-21 06:55:08,195 - lava_callback - INFO - Starting async job submit for node 69e47c41355196461733a939 2026-04-21 06:55:08,196 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:55:08,221 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47c41355196461733a939 HTTP/1.1" 200 None 2026-04-21 06:55:08,221 - lava_callback - DEBUG - Processing node 69e47c41355196461733a939: baseline-arm-clabbe 2026-04-21 06:55:08,222 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47c41355196461733a939 2026-04-21 06:55:08,222 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:55:08,237 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47c41355196461733a939 HTTP/1.1" 200 None 2026-04-21 06:55:08,238 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:55:08,251 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 06:55:08,252 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:55:08,277 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47c41355196461733a939 HTTP/1.1" 200 None 2026-04-21 06:55:08,279 - lava_callback - ERROR - Error processing callback for node 69e47c41355196461733a939: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:45956 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45968 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60078 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60082 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45954 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:55:45,023 - lava_callback - INFO - Received callback for node 69e47c64355196461733a949 INFO: 172.20.0.1:42510 - "POST /node/69e47c64355196461733a949 HTTP/1.0" 202 Accepted 2026-04-21 06:55:45,033 - lava_callback - INFO - Starting async job submit for node 69e47c64355196461733a949 2026-04-21 06:55:45,034 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:55:45,057 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47c64355196461733a949 HTTP/1.1" 200 None 2026-04-21 06:55:45,058 - lava_callback - DEBUG - Processing node 69e47c64355196461733a949: baseline-arm-baylibre 2026-04-21 06:55:45,059 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47c64355196461733a949 2026-04-21 06:55:45,059 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:55:45,083 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47c64355196461733a949 HTTP/1.1" 200 None 2026-04-21 06:55:45,085 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:55:45,099 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 06:55:45,101 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:55:45,120 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47c64355196461733a949 HTTP/1.1" 200 None 2026-04-21 06:55:45,121 - lava_callback - ERROR - Error processing callback for node 69e47c64355196461733a949: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:42524 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40904 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55356 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51690 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51706 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47942 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47950 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:56:28,299 - lava_callback - INFO - Received callback for node 69e47c8b355196461733a959 INFO: 172.20.0.1:47962 - "POST /node/69e47c8b355196461733a959 HTTP/1.0" 202 Accepted 2026-04-21 06:56:28,304 - lava_callback - INFO - Starting async job submit for node 69e47c8b355196461733a959 2026-04-21 06:56:28,305 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:56:28,327 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47c8b355196461733a959 HTTP/1.1" 200 None 2026-04-21 06:56:28,328 - lava_callback - DEBUG - Processing node 69e47c8b355196461733a959: baseline-arm-clabbe 2026-04-21 06:56:28,329 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47c8b355196461733a959 2026-04-21 06:56:28,329 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:56:28,361 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47c8b355196461733a959 HTTP/1.1" 200 None 2026-04-21 06:56:28,363 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:56:28,379 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 06:56:28,380 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:56:28,397 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47c8b355196461733a959 HTTP/1.1" 200 None 2026-04-21 06:56:28,399 - lava_callback - ERROR - Error processing callback for node 69e47c8b355196461733a959: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54956 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58796 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:56:56,124 - lava_callback - INFO - Received callback for node 69e71d4560f5826562c17881 INFO: 172.20.0.1:60718 - "POST /node/69e71d4560f5826562c17881 HTTP/1.0" 202 Accepted 2026-04-21 06:56:56,130 - lava_callback - INFO - Starting async job submit for node 69e71d4560f5826562c17881 2026-04-21 06:56:56,166 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:56:56,180 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e71d4560f5826562c17881 HTTP/1.1" 200 None 2026-04-21 06:56:56,181 - lava_callback - DEBUG - Processing node 69e71d4560f5826562c17881: baseline-arm64-broonie 2026-04-21 06:56:56,424 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.20.0.1:60728 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:56:56,551 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 06:56:56,552 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e71d4560f5826562c17881/log.txt.gz 2026-04-21 06:56:56,695 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 06:56:57,014 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 06:56:57,016 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e71d4560f5826562c17881/lava-logs.yaml 2026-04-21 06:56:57,021 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 06:56:57,129 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 06:56:57,131 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e71d4560f5826562c17881/lava_callback.json.gz 2026-04-21 06:56:57,167 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:56:57,181 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e71d4560f5826562c17881 HTTP/1.1" 200 None 2026-04-21 06:56:57,182 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:56:57,195 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e66a77dc041e577b7785d5 HTTP/1.1" 200 None 2026-04-21 06:56:57,196 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:56:57,214 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e71d4560f5826562c17881 HTTP/1.1" 200 None 2026-04-21 06:56:57,295 - lava_callback - INFO - Completed processing callback for node 69e71d4560f5826562c17881 INFO: 172.20.0.1:56842 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:57:08,258 - lava_callback - INFO - Received callback for node 69e47caf355196461733a96a INFO: 172.20.0.1:56850 - "POST /node/69e47caf355196461733a96a HTTP/1.0" 202 Accepted 2026-04-21 06:57:08,264 - lava_callback - INFO - Starting async job submit for node 69e47caf355196461733a96a 2026-04-21 06:57:08,264 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:57:08,278 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47caf355196461733a96a HTTP/1.1" 200 None 2026-04-21 06:57:08,279 - lava_callback - DEBUG - Processing node 69e47caf355196461733a96a: baseline-arm-clabbe 2026-04-21 06:57:08,279 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47caf355196461733a96a 2026-04-21 06:57:08,280 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:57:08,292 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47caf355196461733a96a HTTP/1.1" 200 None 2026-04-21 06:57:08,294 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:57:08,307 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 06:57:08,308 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:57:08,325 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47caf355196461733a96a HTTP/1.1" 200 None 2026-04-21 06:57:08,327 - lava_callback - ERROR - Error processing callback for node 69e47caf355196461733a96a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:58972 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 06:57:16,291 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:57:16,306 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:58978 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:57:25,094 - lava_callback - INFO - Received callback for node 69e47cd2355196461733a977 INFO: 172.20.0.1:59128 - "POST /node/69e47cd2355196461733a977 HTTP/1.0" 202 Accepted 2026-04-21 06:57:25,100 - lava_callback - INFO - Starting async job submit for node 69e47cd2355196461733a977 2026-04-21 06:57:25,101 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:57:25,114 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47cd2355196461733a977 HTTP/1.1" 200 None 2026-04-21 06:57:25,115 - lava_callback - DEBUG - Processing node 69e47cd2355196461733a977: baseline-arm-baylibre 2026-04-21 06:57:25,115 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47cd2355196461733a977 2026-04-21 06:57:25,116 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:57:25,128 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47cd2355196461733a977 HTTP/1.1" 200 None 2026-04-21 06:57:25,130 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:57:25,144 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 06:57:25,146 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:57:25,165 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47cd2355196461733a977 HTTP/1.1" 200 None 2026-04-21 06:57:25,166 - lava_callback - ERROR - Error processing callback for node 69e47cd2355196461733a977: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:59134 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59136 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45550 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60920 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41578 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58518 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:58:08,339 - lava_callback - INFO - Received callback for node 69e47cf8355196461733a983 INFO: 172.20.0.1:58524 - "POST /node/69e47cf8355196461733a983 HTTP/1.0" 202 Accepted 2026-04-21 06:58:08,344 - lava_callback - INFO - Starting async job submit for node 69e47cf8355196461733a983 2026-04-21 06:58:08,345 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:58:08,359 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47cf8355196461733a983 HTTP/1.1" 200 None 2026-04-21 06:58:08,360 - lava_callback - DEBUG - Processing node 69e47cf8355196461733a983: baseline-arm-clabbe 2026-04-21 06:58:08,360 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47cf8355196461733a983 2026-04-21 06:58:08,360 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:58:08,374 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47cf8355196461733a983 HTTP/1.1" 200 None 2026-04-21 06:58:08,375 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:58:08,387 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 06:58:08,389 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:58:08,407 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47cf8355196461733a983 HTTP/1.1" 200 None 2026-04-21 06:58:08,409 - lava_callback - ERROR - Error processing callback for node 69e47cf8355196461733a983: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49460 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49474 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37866 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37868 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57522 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43750 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:58:48,332 - lava_callback - INFO - Received callback for node 69e47d1d355196461733a997 INFO: 172.20.0.1:43760 - "POST /node/69e47d1d355196461733a997 HTTP/1.0" 202 Accepted 2026-04-21 06:58:48,337 - lava_callback - INFO - Starting async job submit for node 69e47d1d355196461733a997 2026-04-21 06:58:48,338 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:58:48,355 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47d1d355196461733a997 HTTP/1.1" 200 None 2026-04-21 06:58:48,356 - lava_callback - DEBUG - Processing node 69e47d1d355196461733a997: baseline-arm-clabbe 2026-04-21 06:58:48,357 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47d1d355196461733a997 2026-04-21 06:58:48,357 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:58:48,371 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47d1d355196461733a997 HTTP/1.1" 200 None 2026-04-21 06:58:48,373 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:58:48,391 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 06:58:48,393 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:58:48,413 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47d1d355196461733a997 HTTP/1.1" 200 None 2026-04-21 06:58:48,414 - lava_callback - ERROR - Error processing callback for node 69e47d1d355196461733a997: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:45702 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58838 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43096 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43106 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52068 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52084 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:59:28,497 - lava_callback - INFO - Received callback for node 69e47d41355196461733a9a7 INFO: 172.20.0.1:52098 - "POST /node/69e47d41355196461733a9a7 HTTP/1.0" 202 Accepted 2026-04-21 06:59:28,502 - lava_callback - INFO - Starting async job submit for node 69e47d41355196461733a9a7 2026-04-21 06:59:28,504 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:59:28,517 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47d41355196461733a9a7 HTTP/1.1" 200 None 2026-04-21 06:59:28,518 - lava_callback - DEBUG - Processing node 69e47d41355196461733a9a7: baseline-arm-clabbe 2026-04-21 06:59:28,518 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47d41355196461733a9a7 2026-04-21 06:59:28,519 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:59:28,537 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47d41355196461733a9a7 HTTP/1.1" 200 None 2026-04-21 06:59:28,538 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:59:28,551 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 06:59:28,553 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:59:28,570 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47d41355196461733a9a7 HTTP/1.1" 200 None 2026-04-21 06:59:28,571 - lava_callback - ERROR - Error processing callback for node 69e47d41355196461733a9a7: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-04-21 06:59:36,182 - lava_callback - INFO - Received callback for node 69e71d9960f5826562c1789e INFO: 172.20.0.1:44514 - "POST /node/69e71d9960f5826562c1789e HTTP/1.0" 202 Accepted 2026-04-21 06:59:36,188 - lava_callback - INFO - Starting async job submit for node 69e71d9960f5826562c1789e 2026-04-21 06:59:36,230 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:59:36,244 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e71d9960f5826562c1789e HTTP/1.1" 200 None 2026-04-21 06:59:36,245 - lava_callback - DEBUG - Processing node 69e71d9960f5826562c1789e: baseline-arm64-broonie 2026-04-21 06:59:36,459 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.20.0.1:44528 - "GET / HTTP/1.0" 200 OK 2026-04-21 06:59:36,716 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 06:59:36,718 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e71d9960f5826562c1789e/log.txt.gz 2026-04-21 06:59:36,874 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 06:59:37,055 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 06:59:37,057 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e71d9960f5826562c1789e/lava-logs.yaml 2026-04-21 06:59:37,062 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 06:59:37,329 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 06:59:37,331 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e71d9960f5826562c1789e/lava_callback.json.gz 2026-04-21 06:59:37,367 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:59:37,381 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e71d9960f5826562c1789e HTTP/1.1" 200 None 2026-04-21 06:59:37,383 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:59:37,397 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6522047863bb428fe4af5 HTTP/1.1" 200 None 2026-04-21 06:59:37,398 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:59:37,416 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e71d9960f5826562c1789e HTTP/1.1" 200 None 2026-04-21 06:59:37,488 - lava_callback - INFO - Completed processing callback for node 69e71d9960f5826562c1789e 2026-04-21 06:59:46,309 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 06:59:46,329 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:40210 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49170 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36880 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:00:08,550 - lava_callback - INFO - Received callback for node 69e47d67355196461733a9b6 INFO: 172.20.0.1:36884 - "POST /node/69e47d67355196461733a9b6 HTTP/1.0" 202 Accepted 2026-04-21 07:00:08,555 - lava_callback - INFO - Starting async job submit for node 69e47d67355196461733a9b6 2026-04-21 07:00:08,556 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:00:08,570 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47d67355196461733a9b6 HTTP/1.1" 200 None 2026-04-21 07:00:08,571 - lava_callback - DEBUG - Processing node 69e47d67355196461733a9b6: baseline-arm-clabbe 2026-04-21 07:00:08,572 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47d67355196461733a9b6 2026-04-21 07:00:08,572 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:00:08,585 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47d67355196461733a9b6 HTTP/1.1" 200 None 2026-04-21 07:00:08,587 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:00:08,610 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 07:00:08,611 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:00:08,630 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47d67355196461733a9b6 HTTP/1.1" 200 None 2026-04-21 07:00:08,632 - lava_callback - ERROR - Error processing callback for node 69e47d67355196461733a9b6: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:60194 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60200 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:00:25,259 - lava_callback - INFO - Received callback for node 69e47d86355196461733a9c3 INFO: 172.20.0.1:51228 - "POST /node/69e47d86355196461733a9c3 HTTP/1.0" 202 Accepted 2026-04-21 07:00:25,265 - lava_callback - INFO - Starting async job submit for node 69e47d86355196461733a9c3 2026-04-21 07:00:25,266 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:00:25,288 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47d86355196461733a9c3 HTTP/1.1" 200 None 2026-04-21 07:00:25,289 - lava_callback - DEBUG - Processing node 69e47d86355196461733a9c3: baseline-arm-baylibre 2026-04-21 07:00:25,289 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47d86355196461733a9c3 2026-04-21 07:00:25,290 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:00:25,303 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47d86355196461733a9c3 HTTP/1.1" 200 None 2026-04-21 07:00:25,304 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:00:25,317 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 07:00:25,318 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:00:25,336 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47d86355196461733a9c3 HTTP/1.1" 200 None 2026-04-21 07:00:25,337 - lava_callback - ERROR - Error processing callback for node 69e47d86355196461733a9c3: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:51244 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51252 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42306 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58218 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36452 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57958 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:01:08,555 - lava_callback - INFO - Received callback for node 69e47db0355196461733a9d2 INFO: 172.20.0.1:57968 - "POST /node/69e47db0355196461733a9d2 HTTP/1.0" 202 Accepted 2026-04-21 07:01:08,560 - lava_callback - INFO - Starting async job submit for node 69e47db0355196461733a9d2 2026-04-21 07:01:08,561 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:01:08,588 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47db0355196461733a9d2 HTTP/1.1" 200 None 2026-04-21 07:01:08,589 - lava_callback - DEBUG - Processing node 69e47db0355196461733a9d2: baseline-arm-clabbe 2026-04-21 07:01:08,589 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47db0355196461733a9d2 2026-04-21 07:01:08,590 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:01:08,603 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47db0355196461733a9d2 HTTP/1.1" 200 None 2026-04-21 07:01:08,605 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:01:08,630 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 07:01:08,632 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:01:08,649 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47db0355196461733a9d2 HTTP/1.1" 200 None 2026-04-21 07:01:08,650 - lava_callback - ERROR - Error processing callback for node 69e47db0355196461733a9d2: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47922 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47932 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53670 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53672 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60392 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:01:45,378 - lava_callback - INFO - Received callback for node 69e47dcf355196461733a9e2 INFO: 172.20.0.1:51552 - "POST /node/69e47dcf355196461733a9e2 HTTP/1.0" 202 Accepted 2026-04-21 07:01:45,384 - lava_callback - INFO - Starting async job submit for node 69e47dcf355196461733a9e2 2026-04-21 07:01:45,385 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:01:45,403 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47dcf355196461733a9e2 HTTP/1.1" 200 None 2026-04-21 07:01:45,404 - lava_callback - DEBUG - Processing node 69e47dcf355196461733a9e2: baseline-arm-baylibre 2026-04-21 07:01:45,405 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47dcf355196461733a9e2 2026-04-21 07:01:45,405 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:01:45,418 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47dcf355196461733a9e2 HTTP/1.1" 200 None 2026-04-21 07:01:45,420 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:01:45,433 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 07:01:45,434 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:01:45,453 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47dcf355196461733a9e2 HTTP/1.1" 200 None 2026-04-21 07:01:45,454 - lava_callback - ERROR - Error processing callback for node 69e47dcf355196461733a9e2: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:51562 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46082 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54008 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56064 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:56072 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:02:25,337 - lava_callback - INFO - Received callback for node 69e47df3355196461733a9f1 INFO: 172.20.0.1:49650 - "POST /node/69e47df3355196461733a9f1 HTTP/1.0" 202 Accepted 2026-04-21 07:02:25,343 - lava_callback - INFO - Starting async job submit for node 69e47df3355196461733a9f1 2026-04-21 07:02:25,344 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:02:25,358 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47df3355196461733a9f1 HTTP/1.1" 200 None 2026-04-21 07:02:25,359 - lava_callback - DEBUG - Processing node 69e47df3355196461733a9f1: baseline-arm-baylibre 2026-04-21 07:02:25,359 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47df3355196461733a9f1 2026-04-21 07:02:25,359 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:02:25,372 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47df3355196461733a9f1 HTTP/1.1" 200 None 2026-04-21 07:02:25,373 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:02:25,394 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 07:02:25,395 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:02:25,413 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47df3355196461733a9f1 HTTP/1.1" 200 None 2026-04-21 07:02:25,414 - lava_callback - ERROR - Error processing callback for node 69e47df3355196461733a9f1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49666 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49674 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54344 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54996 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60536 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38306 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:03:08,577 - lava_callback - INFO - Received callback for node 69e47e1b355196461733aa02 INFO: 172.20.0.1:38318 - "POST /node/69e47e1b355196461733aa02 HTTP/1.0" 202 Accepted 2026-04-21 07:03:08,583 - lava_callback - INFO - Starting async job submit for node 69e47e1b355196461733aa02 2026-04-21 07:03:08,584 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:03:08,598 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47e1b355196461733aa02 HTTP/1.1" 200 None 2026-04-21 07:03:08,599 - lava_callback - DEBUG - Processing node 69e47e1b355196461733aa02: baseline-arm-clabbe 2026-04-21 07:03:08,599 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47e1b355196461733aa02 2026-04-21 07:03:08,600 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:03:08,613 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47e1b355196461733aa02 HTTP/1.1" 200 None 2026-04-21 07:03:08,614 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:03:08,626 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 07:03:08,628 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:03:08,657 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47e1b355196461733aa02 HTTP/1.1" 200 None 2026-04-21 07:03:08,659 - lava_callback - ERROR - Error processing callback for node 69e47e1b355196461733aa02: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:36102 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:36118 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45038 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45042 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41190 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34408 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:03:48,569 - lava_callback - INFO - Received callback for node 69e47e41355196461733aa12 INFO: 172.20.0.1:34416 - "POST /node/69e47e41355196461733aa12 HTTP/1.0" 202 Accepted 2026-04-21 07:03:48,575 - lava_callback - INFO - Starting async job submit for node 69e47e41355196461733aa12 2026-04-21 07:03:48,576 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:03:48,598 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47e41355196461733aa12 HTTP/1.1" 200 None 2026-04-21 07:03:48,599 - lava_callback - DEBUG - Processing node 69e47e41355196461733aa12: baseline-arm-clabbe 2026-04-21 07:03:48,600 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47e41355196461733aa12 2026-04-21 07:03:48,600 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:03:48,614 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47e41355196461733aa12 HTTP/1.1" 200 None 2026-04-21 07:03:48,615 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:03:48,632 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 07:03:48,633 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:03:48,651 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47e41355196461733aa12 HTTP/1.1" 200 None 2026-04-21 07:03:48,652 - lava_callback - ERROR - Error processing callback for node 69e47e41355196461733aa12: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:37684 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:04:05,496 - lava_callback - INFO - Received callback for node 69e47e62355196461733aa1e INFO: 172.20.0.1:47912 - "POST /node/69e47e62355196461733aa1e HTTP/1.0" 202 Accepted 2026-04-21 07:04:05,502 - lava_callback - INFO - Starting async job submit for node 69e47e62355196461733aa1e 2026-04-21 07:04:05,503 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:04:05,524 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47e62355196461733aa1e HTTP/1.1" 200 None 2026-04-21 07:04:05,525 - lava_callback - DEBUG - Processing node 69e47e62355196461733aa1e: baseline-arm-baylibre 2026-04-21 07:04:05,526 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47e62355196461733aa1e 2026-04-21 07:04:05,527 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:04:05,540 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47e62355196461733aa1e HTTP/1.1" 200 None 2026-04-21 07:04:05,542 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:04:05,562 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 07:04:05,565 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:04:05,606 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47e62355196461733aa1e HTTP/1.1" 200 None 2026-04-21 07:04:05,607 - lava_callback - ERROR - Error processing callback for node 69e47e62355196461733aa1e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47922 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46520 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46534 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59938 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59942 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45784 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:04:45,492 - lava_callback - INFO - Received callback for node 69e47e88355196461733aa2e INFO: 172.20.0.1:45714 - "POST /node/69e47e88355196461733aa2e HTTP/1.0" 202 Accepted 2026-04-21 07:04:45,497 - lava_callback - INFO - Starting async job submit for node 69e47e88355196461733aa2e 2026-04-21 07:04:45,498 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:04:45,516 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47e88355196461733aa2e HTTP/1.1" 200 None 2026-04-21 07:04:45,517 - lava_callback - DEBUG - Processing node 69e47e88355196461733aa2e: baseline-arm-baylibre 2026-04-21 07:04:45,517 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47e88355196461733aa2e 2026-04-21 07:04:45,518 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:04:45,531 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47e88355196461733aa2e HTTP/1.1" 200 None 2026-04-21 07:04:45,533 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:04:45,545 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 07:04:45,547 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:04:45,563 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47e88355196461733aa2e HTTP/1.1" 200 None 2026-04-21 07:04:45,564 - lava_callback - ERROR - Error processing callback for node 69e47e88355196461733aa2e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:45724 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:04:56,073 - lava_callback - INFO - Received callback for node 69e71de860f5826562c178c2 INFO: 172.20.0.1:60984 - "POST /node/69e71de860f5826562c178c2 HTTP/1.0" 202 Accepted 2026-04-21 07:04:56,079 - lava_callback - INFO - Starting async job submit for node 69e71de860f5826562c178c2 2026-04-21 07:04:56,115 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:04:56,129 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e71de860f5826562c178c2 HTTP/1.1" 200 None 2026-04-21 07:04:56,130 - lava_callback - DEBUG - Processing node 69e71de860f5826562c178c2: baseline-arm64-broonie 2026-04-21 07:04:56,360 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 07:04:56,476 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 07:04:56,478 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e71de860f5826562c178c2/log.txt.gz INFO: 172.20.0.1:60996 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:04:56,605 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 07:04:56,925 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 07:04:56,927 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e71de860f5826562c178c2/lava-logs.yaml 2026-04-21 07:04:56,932 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 07:04:57,038 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 07:04:57,039 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e71de860f5826562c178c2/lava_callback.json.gz 2026-04-21 07:04:57,075 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:04:57,110 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e71de860f5826562c178c2 HTTP/1.1" 200 None 2026-04-21 07:04:57,112 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:04:57,125 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6c22160f5826562c1509a HTTP/1.1" 200 None 2026-04-21 07:04:57,126 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:04:57,144 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e71de860f5826562c178c2 HTTP/1.1" 200 None 2026-04-21 07:04:57,221 - lava_callback - INFO - Completed processing callback for node 69e71de860f5826562c178c2 INFO: 172.20.0.1:47466 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56902 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 07:05:16,332 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:05:16,348 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:56912 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48992 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49004 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:05:28,737 - lava_callback - INFO - Received callback for node 69e47eaf355196461733aa42 INFO: 172.20.0.1:49014 - "POST /node/69e47eaf355196461733aa42 HTTP/1.0" 202 Accepted 2026-04-21 07:05:28,743 - lava_callback - INFO - Starting async job submit for node 69e47eaf355196461733aa42 2026-04-21 07:05:28,744 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:05:28,761 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47eaf355196461733aa42 HTTP/1.1" 200 None 2026-04-21 07:05:28,762 - lava_callback - DEBUG - Processing node 69e47eaf355196461733aa42: baseline-arm-clabbe 2026-04-21 07:05:28,762 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47eaf355196461733aa42 2026-04-21 07:05:28,763 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:05:28,776 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47eaf355196461733aa42 HTTP/1.1" 200 None 2026-04-21 07:05:28,777 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:05:28,795 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 07:05:28,797 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:05:28,814 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47eaf355196461733aa42 HTTP/1.1" 200 None 2026-04-21 07:05:28,815 - lava_callback - ERROR - Error processing callback for node 69e47eaf355196461733aa42: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:43206 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35746 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58764 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44994 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:06:08,736 - lava_callback - INFO - Received callback for node 69e47ed4355196461733aa4e INFO: 172.20.0.1:44998 - "POST /node/69e47ed4355196461733aa4e HTTP/1.0" 202 Accepted 2026-04-21 07:06:08,742 - lava_callback - INFO - Starting async job submit for node 69e47ed4355196461733aa4e 2026-04-21 07:06:08,743 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:06:08,767 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47ed4355196461733aa4e HTTP/1.1" 200 None 2026-04-21 07:06:08,768 - lava_callback - DEBUG - Processing node 69e47ed4355196461733aa4e: baseline-arm-clabbe 2026-04-21 07:06:08,768 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47ed4355196461733aa4e 2026-04-21 07:06:08,769 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:06:08,782 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47ed4355196461733aa4e HTTP/1.1" 200 None 2026-04-21 07:06:08,783 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:06:08,795 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 07:06:08,797 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:06:08,814 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47ed4355196461733aa4e HTTP/1.1" 200 None 2026-04-21 07:06:08,815 - lava_callback - ERROR - Error processing callback for node 69e47ed4355196461733aa4e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:39860 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39872 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40420 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40430 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43294 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:06:45,593 - lava_callback - INFO - Received callback for node 69e47ef6355196461733aa5c INFO: 172.20.0.1:46394 - "POST /node/69e47ef6355196461733aa5c HTTP/1.0" 202 Accepted 2026-04-21 07:06:45,598 - lava_callback - INFO - Starting async job submit for node 69e47ef6355196461733aa5c 2026-04-21 07:06:45,600 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:06:45,615 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47ef6355196461733aa5c HTTP/1.1" 200 None 2026-04-21 07:06:45,616 - lava_callback - DEBUG - Processing node 69e47ef6355196461733aa5c: baseline-arm-baylibre 2026-04-21 07:06:45,616 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47ef6355196461733aa5c 2026-04-21 07:06:45,617 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:06:45,647 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47ef6355196461733aa5c HTTP/1.1" 200 None 2026-04-21 07:06:45,648 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:06:45,662 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 07:06:45,663 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:06:45,681 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47ef6355196461733aa5c HTTP/1.1" 200 None 2026-04-21 07:06:45,682 - lava_callback - ERROR - Error processing callback for node 69e47ef6355196461733aa5c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:46396 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35704 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35292 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43806 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43812 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52244 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52248 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:07:28,851 - lava_callback - INFO - Received callback for node 69e47f1b355196461733aa6e INFO: 172.20.0.1:52254 - "POST /node/69e47f1b355196461733aa6e HTTP/1.0" 202 Accepted 2026-04-21 07:07:28,857 - lava_callback - INFO - Starting async job submit for node 69e47f1b355196461733aa6e 2026-04-21 07:07:28,857 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:07:28,872 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47f1b355196461733aa6e HTTP/1.1" 200 None 2026-04-21 07:07:28,872 - lava_callback - DEBUG - Processing node 69e47f1b355196461733aa6e: baseline-arm-clabbe 2026-04-21 07:07:28,873 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47f1b355196461733aa6e 2026-04-21 07:07:28,873 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:07:28,886 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47f1b355196461733aa6e HTTP/1.1" 200 None 2026-04-21 07:07:28,888 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:07:28,907 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 07:07:28,908 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:07:28,931 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47f1b355196461733aa6e HTTP/1.1" 200 None 2026-04-21 07:07:28,933 - lava_callback - ERROR - Error processing callback for node 69e47f1b355196461733aa6e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-04-21 07:07:36,515 - lava_callback - INFO - Received callback for node 69e71e3660f5826562c178ea INFO: 172.20.0.1:32768 - "POST /node/69e71e3660f5826562c178ea HTTP/1.0" 202 Accepted 2026-04-21 07:07:36,521 - lava_callback - INFO - Starting async job submit for node 69e71e3660f5826562c178ea INFO: 172.20.0.1:32774 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:07:36,577 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:07:36,591 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e71e3660f5826562c178ea HTTP/1.1" 200 None 2026-04-21 07:07:36,591 - lava_callback - DEBUG - Processing node 69e71e3660f5826562c178ea: baseline-arm64-broonie 2026-04-21 07:07:36,836 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 07:07:36,972 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 07:07:36,974 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e71e3660f5826562c178ea/log.txt.gz 2026-04-21 07:07:37,098 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 07:07:37,405 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 07:07:37,407 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e71e3660f5826562c178ea/lava-logs.yaml 2026-04-21 07:07:37,413 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 07:07:37,511 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 07:07:37,513 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e71e3660f5826562c178ea/lava_callback.json.gz 2026-04-21 07:07:37,548 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:07:37,571 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e71e3660f5826562c178ea HTTP/1.1" 200 None 2026-04-21 07:07:37,572 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:07:37,585 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e66a77dc041e577b7785d5 HTTP/1.1" 200 None 2026-04-21 07:07:37,587 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:07:37,604 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e71e3660f5826562c178ea HTTP/1.1" 200 None 2026-04-21 07:07:37,678 - lava_callback - INFO - Completed processing callback for node 69e71e3660f5826562c178ea 2026-04-21 07:07:46,350 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:07:46,366 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:55234 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:07:48,800 - lava_callback - INFO - Received callback for node 69e47f41355196461733aa7c INFO: 172.20.0.1:55248 - "POST /node/69e47f41355196461733aa7c HTTP/1.0" 202 Accepted 2026-04-21 07:07:48,805 - lava_callback - INFO - Starting async job submit for node 69e47f41355196461733aa7c 2026-04-21 07:07:48,806 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:07:48,820 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47f41355196461733aa7c HTTP/1.1" 200 None 2026-04-21 07:07:48,821 - lava_callback - DEBUG - Processing node 69e47f41355196461733aa7c: baseline-arm-clabbe 2026-04-21 07:07:48,821 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47f41355196461733aa7c 2026-04-21 07:07:48,822 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:07:48,839 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47f41355196461733aa7c HTTP/1.1" 200 None 2026-04-21 07:07:48,840 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:07:48,852 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 07:07:48,854 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:07:48,872 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47f41355196461733aa7c HTTP/1.1" 200 None 2026-04-21 07:07:48,873 - lava_callback - ERROR - Error processing callback for node 69e47f41355196461733aa7c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:44900 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47324 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58176 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58178 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39810 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39824 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:08:28,846 - lava_callback - INFO - Received callback for node 69e47f66355196461733aa8b INFO: 172.20.0.1:39840 - "POST /node/69e47f66355196461733aa8b HTTP/1.0" 202 Accepted 2026-04-21 07:08:28,851 - lava_callback - INFO - Starting async job submit for node 69e47f66355196461733aa8b 2026-04-21 07:08:28,852 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:08:28,866 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47f66355196461733aa8b HTTP/1.1" 200 None 2026-04-21 07:08:28,867 - lava_callback - DEBUG - Processing node 69e47f66355196461733aa8b: baseline-arm-clabbe 2026-04-21 07:08:28,867 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47f66355196461733aa8b 2026-04-21 07:08:28,868 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:08:28,882 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47f66355196461733aa8b HTTP/1.1" 200 None 2026-04-21 07:08:28,883 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:08:28,897 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 07:08:28,898 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:08:28,916 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47f66355196461733aa8b HTTP/1.1" 200 None 2026-04-21 07:08:28,917 - lava_callback - ERROR - Error processing callback for node 69e47f66355196461733aa8b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:42546 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50184 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53682 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54494 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:09:08,860 - lava_callback - INFO - Received callback for node 69e47f8c355196461733aa99 INFO: 172.20.0.1:54504 - "POST /node/69e47f8c355196461733aa99 HTTP/1.0" 202 Accepted 2026-04-21 07:09:08,865 - lava_callback - INFO - Starting async job submit for node 69e47f8c355196461733aa99 2026-04-21 07:09:08,866 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:09:08,880 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47f8c355196461733aa99 HTTP/1.1" 200 None 2026-04-21 07:09:08,881 - lava_callback - DEBUG - Processing node 69e47f8c355196461733aa99: baseline-arm-clabbe 2026-04-21 07:09:08,881 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47f8c355196461733aa99 2026-04-21 07:09:08,882 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:09:08,894 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47f8c355196461733aa99 HTTP/1.1" 200 None 2026-04-21 07:09:08,895 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:09:08,908 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 07:09:08,910 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:09:08,951 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47f8c355196461733aa99 HTTP/1.1" 200 None 2026-04-21 07:09:08,953 - lava_callback - ERROR - Error processing callback for node 69e47f8c355196461733aa99: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:37834 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37842 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45342 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45354 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58414 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60338 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:09:48,917 - lava_callback - INFO - Received callback for node 69e47fb1355196461733aaa8 INFO: 172.20.0.1:60354 - "POST /node/69e47fb1355196461733aaa8 HTTP/1.0" 202 Accepted 2026-04-21 07:09:48,922 - lava_callback - INFO - Starting async job submit for node 69e47fb1355196461733aaa8 2026-04-21 07:09:48,923 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:09:48,936 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47fb1355196461733aaa8 HTTP/1.1" 200 None 2026-04-21 07:09:48,937 - lava_callback - DEBUG - Processing node 69e47fb1355196461733aaa8: baseline-arm-clabbe 2026-04-21 07:09:48,937 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47fb1355196461733aaa8 2026-04-21 07:09:48,938 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:09:48,950 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47fb1355196461733aaa8 HTTP/1.1" 200 None 2026-04-21 07:09:48,952 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:09:48,965 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 07:09:48,967 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:09:48,989 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47fb1355196461733aaa8 HTTP/1.1" 200 None 2026-04-21 07:09:48,990 - lava_callback - ERROR - Error processing callback for node 69e47fb1355196461733aaa8: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:33186 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58304 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47664 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47674 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38182 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38188 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:10:28,932 - lava_callback - INFO - Received callback for node 69e47fd6355196461733aab6 INFO: 172.20.0.1:38198 - "POST /node/69e47fd6355196461733aab6 HTTP/1.0" 202 Accepted 2026-04-21 07:10:28,939 - lava_callback - INFO - Starting async job submit for node 69e47fd6355196461733aab6 2026-04-21 07:10:28,940 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:10:28,968 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47fd6355196461733aab6 HTTP/1.1" 200 None 2026-04-21 07:10:28,969 - lava_callback - DEBUG - Processing node 69e47fd6355196461733aab6: baseline-arm-clabbe 2026-04-21 07:10:28,969 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47fd6355196461733aab6 2026-04-21 07:10:28,970 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:10:28,983 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47fd6355196461733aab6 HTTP/1.1" 200 None 2026-04-21 07:10:28,984 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:10:28,997 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 07:10:28,998 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:10:29,016 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47fd6355196461733aab6 HTTP/1.1" 200 None 2026-04-21 07:10:29,017 - lava_callback - ERROR - Error processing callback for node 69e47fd6355196461733aab6: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:45444 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41050 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49926 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42418 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:11:09,022 - lava_callback - INFO - Received callback for node 69e47ffb355196461733aac5 INFO: 172.20.0.1:42430 - "POST /node/69e47ffb355196461733aac5 HTTP/1.0" 202 Accepted 2026-04-21 07:11:09,028 - lava_callback - INFO - Starting async job submit for node 69e47ffb355196461733aac5 2026-04-21 07:11:09,028 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:11:09,061 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47ffb355196461733aac5 HTTP/1.1" 200 None 2026-04-21 07:11:09,062 - lava_callback - DEBUG - Processing node 69e47ffb355196461733aac5: baseline-arm-clabbe 2026-04-21 07:11:09,062 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e47ffb355196461733aac5 2026-04-21 07:11:09,063 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:11:09,076 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e47ffb355196461733aac5 HTTP/1.1" 200 None 2026-04-21 07:11:09,078 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:11:09,091 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 07:11:09,092 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:11:09,110 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e47ffb355196461733aac5 HTTP/1.1" 200 None 2026-04-21 07:11:09,111 - lava_callback - ERROR - Error processing callback for node 69e47ffb355196461733aac5: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:55080 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:55084 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56146 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56148 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48288 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39224 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:11:48,967 - lava_callback - INFO - Received callback for node 69e48020355196461733aacf INFO: 172.20.0.1:39240 - "POST /node/69e48020355196461733aacf HTTP/1.0" 202 Accepted 2026-04-21 07:11:48,972 - lava_callback - INFO - Starting async job submit for node 69e48020355196461733aacf 2026-04-21 07:11:48,973 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:11:48,987 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e48020355196461733aacf HTTP/1.1" 200 None 2026-04-21 07:11:48,988 - lava_callback - DEBUG - Processing node 69e48020355196461733aacf: baseline-arm-clabbe 2026-04-21 07:11:48,988 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e48020355196461733aacf 2026-04-21 07:11:48,989 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:11:49,001 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e48020355196461733aacf HTTP/1.1" 200 None 2026-04-21 07:11:49,003 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:11:49,016 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 07:11:49,017 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:11:49,037 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e48020355196461733aacf HTTP/1.1" 200 None 2026-04-21 07:11:49,038 - lava_callback - ERROR - Error processing callback for node 69e48020355196461733aacf: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:56258 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38042 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:12:09,054 - lava_callback - INFO - Received callback for node 69e48045355196461733aadd INFO: 172.20.0.1:38056 - "POST /node/69e48045355196461733aadd HTTP/1.0" 202 Accepted 2026-04-21 07:12:09,059 - lava_callback - INFO - Starting async job submit for node 69e48045355196461733aadd 2026-04-21 07:12:09,060 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:12:09,095 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e48045355196461733aadd HTTP/1.1" 200 None 2026-04-21 07:12:09,096 - lava_callback - DEBUG - Processing node 69e48045355196461733aadd: baseline-arm-clabbe 2026-04-21 07:12:09,097 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69e48045355196461733aadd 2026-04-21 07:12:09,097 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:12:09,111 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e48045355196461733aadd HTTP/1.1" 200 None 2026-04-21 07:12:09,112 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:12:09,125 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ded936f490930782fb6e1e HTTP/1.1" 200 None 2026-04-21 07:12:09,127 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:12:09,179 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e48045355196461733aadd HTTP/1.1" 200 None 2026-04-21 07:12:09,180 - lava_callback - ERROR - Error processing callback for node 69e48045355196461733aadd: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 373, in async_job_submit _emit_callback_telemetry(job_node, job_callback, hierarchy) File "/home/kernelci/pipeline/lava_callback.py", line 393, in _emit_callback_telemetry job_callback.is_infra_error() File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 117, in is_infra_error lava_yaml = self._data["results"]["lava"] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:45436 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45444 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60166 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60182 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39204 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40978 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46276 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47052 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44178 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44184 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:13:18,464 - lava_callback - INFO - Received callback for node 69e71e8d60f5826562c17913 INFO: 172.20.0.1:44188 - "POST /node/69e71e8d60f5826562c17913 HTTP/1.0" 202 Accepted 2026-04-21 07:13:18,470 - lava_callback - INFO - Starting async job submit for node 69e71e8d60f5826562c17913 2026-04-21 07:13:18,506 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:13:18,520 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e71e8d60f5826562c17913 HTTP/1.1" 200 None 2026-04-21 07:13:18,521 - lava_callback - DEBUG - Processing node 69e71e8d60f5826562c17913: baseline-arm64-broonie 2026-04-21 07:13:18,727 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 07:13:18,830 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 07:13:18,832 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e71e8d60f5826562c17913/log.txt.gz 2026-04-21 07:13:18,953 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 07:13:19,212 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 07:13:19,214 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e71e8d60f5826562c17913/lava-logs.yaml 2026-04-21 07:13:19,219 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 07:13:19,314 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 07:13:19,315 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e71e8d60f5826562c17913/lava_callback.json.gz 2026-04-21 07:13:19,352 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:13:19,365 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e71e8d60f5826562c17913 HTTP/1.1" 200 None 2026-04-21 07:13:19,367 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:13:19,383 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e6522047863bb428fe4af5 HTTP/1.1" 200 None 2026-04-21 07:13:19,384 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:13:19,401 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e71e8d60f5826562c17913 HTTP/1.1" 200 None 2026-04-21 07:13:19,474 - lava_callback - INFO - Completed processing callback for node 69e71e8d60f5826562c17913 INFO: 172.20.0.1:60910 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60916 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51988 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:13:46,369 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:13:46,384 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:53156 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56372 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41782 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49968 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49970 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41672 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41684 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53926 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50742 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41924 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38948 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56810 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:56826 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56424 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56428 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34110 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57310 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48572 - "GET / HTTP/1.0" 200 OK 2026-04-21 07:15:58,178 - lava_callback - INFO - Received callback for node 69e71fcb60f5826562c1794d INFO: 172.20.0.1:48580 - "POST /node/69e71fcb60f5826562c1794d HTTP/1.0" 202 Accepted 2026-04-21 07:15:58,184 - lava_callback - INFO - Starting async job submit for node 69e71fcb60f5826562c1794d 2026-04-21 07:15:58,220 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:15:58,234 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e71fcb60f5826562c1794d HTTP/1.1" 200 None 2026-04-21 07:15:58,235 - lava_callback - DEBUG - Processing node 69e71fcb60f5826562c1794d: baseline-arm64-broonie 2026-04-21 07:15:58,460 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 07:15:58,573 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 07:15:58,575 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e71fcb60f5826562c1794d/log.txt.gz 2026-04-21 07:15:58,696 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 07:15:58,857 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 07:15:58,859 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e71fcb60f5826562c1794d/lava-logs.yaml 2026-04-21 07:15:58,864 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-04-21 07:15:58,961 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-04-21 07:15:58,962 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69e71fcb60f5826562c1794d/lava_callback.json.gz 2026-04-21 07:15:58,997 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:15:59,015 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e71fcb60f5826562c1794d HTTP/1.1" 200 None 2026-04-21 07:15:59,016 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:15:59,029 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69e66a77dc041e577b7785d5 HTTP/1.1" 200 None 2026-04-21 07:15:59,031 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:15:59,053 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69e71fcb60f5826562c1794d HTTP/1.1" 200 None 2026-04-21 07:15:59,148 - lava_callback - INFO - Completed processing callback for node 69e71fcb60f5826562c1794d INFO: 172.20.0.1:39010 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48672 - "GET /api/metrics HTTP/1.0" 200 OK 2026-04-21 07:16:16,387 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-04-21 07:16:16,411 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:48684 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45152 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45158 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47770 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54956 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58180 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42976 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47762 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47768 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60940 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60954 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42166 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35906 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39130 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46724 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52036 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52050 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33080 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33088 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56306 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42882 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39868 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53672 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35420 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35422 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48252 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48264 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46176 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41316 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41590 - "GET / HTTP/1.0" 200 OK INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [1]